Author Topic: Lost everyone for 24 hr (Read 1162 times)

dpe.ahoy · « **on:** February 20, 2009, 03:23:24 PM »

For almost a full 24 hr I couldn't get on GBO

. I know cannon6 couldn't either, anyone else? DP

Airsporter · « **Reply #1 on:** February 20, 2009, 03:25:19 PM »

Yeah, its like site disappeared

superdown · « **Reply #2 on:** February 20, 2009, 03:34:11 PM »

I couldn't get here either.

brennemanj83 · « **Reply #3 on:** February 20, 2009, 03:37:43 PM »

Same here

PigBoy Crabshaw · « **Reply #4 on:** February 20, 2009, 03:43:13 PM »

I couldn't get on for 6:30am my time up until a few hours ago when I tried from work. -g

cwlongshot · « **Reply #5 on:** February 20, 2009, 03:46:15 PM »

Me too.. I REALLY MISSED you guys!!!
I was like a little kid, I must have tried twenty five times today!! I even sent Tim a note to be sure you guys didn't toss me out for too many posts in the past month!!!

I'm like a druggie, I need my HANDI fix... BAD!!

CW

LONGTOM · « **Reply #6 on:** February 20, 2009, 03:47:14 PM »

I lost it at around midnight untill about 7:30 this morning and again about 9am till 9:30am.
Lost it again tonight around 7:30 till 7:45.
Matt had a messup with some new software but he & Gregg are getting it taken care of.
Still don't have spell check back yet.

LONGTOM

cbourbeau32 · « **Reply #7 on:** February 20, 2009, 03:49:00 PM »

I have checked off and on all day. It's great to see it back. Charlie

Datil · « **Reply #8 on:** February 20, 2009, 04:19:34 PM »

I couldn't get on all day, from 7:oo AM till after6PM.
Marv

dpe.ahoy · « **Reply #9 on:** February 20, 2009, 04:30:16 PM »

Yep, was kinda lonely with out all you guys. No lies to listen to, no one to tell lies to, it was scary! Hey Longtom, i tink mi spel cek is fin. DP

trotterlg · « **Reply #10 on:** February 20, 2009, 05:45:32 PM »

A near exact 24 hour down time usually means some one hosed up a DNS change, it takes that long to get it fixed, they send out the new DNS data once a day, so if you mess it up then you have to wait. Larry

LONGTOM · « **Reply #11 on:** February 20, 2009, 05:55:29 PM »

Quote from: dpe.ahoy on February 20, 2009, 04:30:16 PM

Yep, was kinda lonely with out all you guys. No lies to listen to, no one to tell lies to, it was scary! Hey Longtom, i tink mi spel cek is fin. DP

Looks like my kind of typing.
I understood it!

LONGTOM

DalesCarpentry · « **Reply #12 on:** February 20, 2009, 06:07:26 PM »

Quote from: trotterlg on February 20, 2009, 05:45:32 PM

A near exact 24 hour down time usually means some one hosed up a DNS change, it takes that long to get it fixed, they send out the new DNS data once a day, so if you mess it up then you have to wait. Larry

No Larry. That was not the problem. Dale

DalesCarpentry · « **Reply #13 on:** February 20, 2009, 06:10:29 PM »

The down time really was between 7 and 9 hours. The site has had problems all day though. Dale

SM Bob · « **Reply #14 on:** February 20, 2009, 06:22:43 PM »

10-4! I was going through withdrawals!

Don't do this to us!

Just kidding!

Robert

trotterlg · « **Reply #15 on:** February 20, 2009, 06:29:49 PM »

When you did a tracert to the site there was no DNS entry registered, this, to me, equals screw up, but then I guess you never know, could just be the PETA people getting even. Larry

GregP42 · « **Reply #16 on:** February 20, 2009, 06:37:56 PM »

Quote from: trotterlg on February 20, 2009, 06:29:49 PM

When you did a tracert to the site there was no DNS entry registered, this, to me, equals screw up, but then I guess you never know, could just be the PETA people getting even. Larry

Larry,

Part of the update Matt did to the server toasted the "named" and "bind" on the server. After figuring that out just had to fix it.

Greg

DalesCarpentry · « **Reply #17 on:** February 20, 2009, 06:38:33 PM »

Quote from: trotterlg on February 20, 2009, 06:29:49 PM

When you did a tracert to the site there was no DNS entry registered, this, to me, equals screw up, but then I guess you never know, could just be the PETA people getting even. Larry

Well it was not even close to 24 hours. Dale

DalesCarpentry · « **Reply #18 on:** February 20, 2009, 06:44:22 PM »

Quote from: trotterlg on February 20, 2009, 06:29:49 PM

When you did a tracert to the site there was no DNS entry registered, this, to me, equals screw up, but then I guess you never know, could just be the PETA people getting even. Larry

What is this a cheap shot?

? The heck with you, know it all.

Dale

Skunk · « **Reply #19 on:** February 20, 2009, 06:49:37 PM »

Quote from: DALESCARPENTRY on February 20, 2009, 06:44:22 PM

Quote from: trotterlg on February 20, 2009, 06:29:49 PM
When you did a tracert to the site there was no DNS entry registered, this, to me, equals screw up, but then I guess you never know, could just be the PETA people getting even. Larry
What is this a cheap shot?? The heck with you, know it all. Dale

OK Dale, put the rum bottle down and back away from the monitor.

DalesCarpentry · « **Reply #20 on:** February 20, 2009, 06:51:54 PM »

Quote from: trotterlg on February 20, 2009, 06:29:49 PM

When you did a tracert to the site there was no DNS entry registered, this, to me, equals screw up, but then I guess you never know, could just be the PETA people getting even. Larry

You have managed to piss me off. You bring up something from another forum that has been gone for a couple weeks. In case you other guys don't know,it was in the Round the potbellied stove. The title was This has to stop. He took a cheap shot here and I know it. Dale

DalesCarpentry · « **Reply #21 on:** February 20, 2009, 06:53:35 PM »

Quote from: Skunk on February 20, 2009, 06:49:37 PM

Quote from: DALESCARPENTRY on February 20, 2009, 06:44:22 PM
Quote from: trotterlg on February 20, 2009, 06:29:49 PM
When you did a tracert to the site there was no DNS entry registered, this, to me, equals screw up, but then I guess you never know, could just be the PETA people getting even. Larry
What is this a cheap shot?? The heck with you, know it all. Dale

OK Dale, put the rum bottle down and back away from the monitor.

I am not even drinking tonight. Dale

Skunk · « **Reply #22 on:** February 20, 2009, 06:55:32 PM »

Ok, well, carry on then.

Matt · « **Reply #23 on:** February 20, 2009, 07:16:07 PM »

Quote from: trotterlg on February 20, 2009, 06:29:49 PM

When you did a tracert to the site there was no DNS entry registered, this, to me, equals screw up, but then I guess you never know, could just be the PETA people getting even. Larry

Yes it was a DNS screw up... bind-chroot got installed and it and plesk which is our CP do not like each other. We run mirrored collocated servers and we handle all DNSing on our own. ns1.gbospace.com and ns2.gbospace.com are our name servers. But the whole problem was that bind-chroot hijacked the dns record path and had named looking in the wrong place for the dns record files. Greg got us focused on the named.conf and rncd.conf files which store the DNS info. Well I started re-writing the named.conf file with the correct info and realized after a dozen errors that the path was wrong... fixed the path and restarted named... 15 seconds later everything was back online.

The site went down for some at or around 7pm thur I started making plans to fly to Dallas where the servers are located around 9pm thur. Around midnight or so the level 3 admin at our hosting provider called and told me the system was in a kernel:panic state so we fought with it for a while and managed to get grub to boot from a backup kernel. Around 3am fri I managed to restore the server or so I thought but after DNS caching started across the net little by little people were losing access. By about noon I think the west coast had lost all access via domain. After another reboot around 5pm I lost access accept by direct IP and I think it was around 7pm that I managed to get it all back online.

So there you have it guys... that’s what happen... and yes it was my fault...

Matt

trotterlg · « **Reply #24 on:** February 20, 2009, 07:22:23 PM »

Darn! Guess the PETA people are off the hook this time, just plain old Human Error. I've done a little of this before Dale, it just happens, I would not know about this if I hadn't screwed up the same way my self. There are tons of online services that you can use to figure out what the problems are, they are free and a lot of fun to use, read up on them and have at it. There are very few secrets on the internet. Larry

DalesCarpentry · « **Reply #25 on:** February 20, 2009, 07:23:15 PM »

Quote from: Matt on February 20, 2009, 07:16:07 PM

Quote from: trotterlg on February 20, 2009, 06:29:49 PM
When you did a tracert to the site there was no DNS entry registered, this, to me, equals screw up, but then I guess you never know, could just be the PETA people getting even. Larry

Yes it was a DNS screw up... bind-chroot got installed and it and plesk which is our CP do not like each other. We run mirrored collocated servers and we handle all DNSing on our own. ns1.gbospace.com and ns2.gbospace.com are our name servers. But the whole problem was that bind-chroot hijacked the dns record path and had named looking in the wrong place for the dns record files. Greg got us focused on the named.conf and rncd.conf files which store the DNS info. Well I started re-writing the named.conf file with the correct info and realized after a dozen errors that the path was wrong... fixed the path and restarted named... 15 seconds later everything was back online.

The site went down for some at or around 7pm thur I started making plans to fly to Dallas where the servers are located around 9pm thur. Around midnight or so the level 3 admin at our hosting provider called and told me the system was in a kernel:panic state so we fought with it for a while and managed to get grub to boot from a backup kernel. Around 3am fri I managed to restore the server or so I thought but after DNS caching started across the net little by little people were losing access. By about noon I think the west coast had lost all access via domain. After another reboot around 5pm I lost access accept by direct IP and I think it was around 7pm that I managed to get it all back online.

So there you have it guys... that’s what happen... and yes it was my fault...

Matt

I have always been told if you never screw up you are not doing anything. I say good job getting us back on line.

Now aboaut thath spel chek ting. Dale

trotterlg · « **Reply #26 on:** February 20, 2009, 07:30:37 PM »

Well Dale, if you had a few drinks in you, you would have been a littel more laid back like me. Larry

Matt · « **Reply #27 on:** February 20, 2009, 07:31:38 PM »

Quote from: DALESCARPENTRY on February 20, 2009, 07:23:15 PM

I have always been told if you never screw up you are not doing anything. I say good job getting us back on line. Now aboaut thath spel chek ting. Dale

working on it

ozarkhillbilly49 · « **Reply #28 on:** February 20, 2009, 07:42:15 PM »

linux is a nice os if you rember some old dos rules ain't it matt.trust me you ain't the first to goof up and i give you my word you won't be the last! just come look at my bench and junk box!!!!!take care all

Hammerspur · « **Reply #29 on:** February 21, 2009, 01:19:11 AM »

I'm just glad everything's back, up n' running...

I was getting to the brink of serious withdrawal!!