More Info On Dead Domain Handling
Posted on September 16, 2006 at 9:39 AM in @earthlink

Note: The below is an archived entry from Earthling, formerly EarthLink's official blog. The blog itself has been decommissioned and is no longer updated, and comments are trackbacks are no longer accepted.

It's early on a Saturday morning and I'm outside of my usual blogging schedule, but I received the following update from the product team and wanted to get it out to Earthling as soon as I could. Several commenters on the previous update have been asking patiently (and a few not so patiently) for more information, so I'm publishing this now and will catch up with everything else -- including any pending comments that may come in over the weekend -- when I'm back in the office on Monday.

"As Dave mentioned last week, we've been working to tune the configuration of dead domain handling to address the issues some EarthLink customers have been experiencing during the rollout. We've been able to mitigate a number of issues, but we've also recognized that some of them such as certain problems with VPN access are not easily addressed through configuration changes in our dead domain system. For customers who are still experiencing issues, we've set aside a pair of servers that will remain outside of our standard DNS system. If you have a need to stay clear of our dead domain handling, just point your DNS servers to:

207.69.188.171 (west coast)

207.69.188.172 (east coast)

If you're west of the Mississippi, use the west coast DNS server as your primary and the east coast as secondary. And vice-versa for those of you in the eastern U.S.

This is a self-service solution, and it creates a mechanism for those users who choose to opt out to do so. These servers have been set aside to resolve problems you may be having, but this isn't something that you'll be able to get customer support help with."

-Ken Womack
Senior Product Manager, Search

Update (9/29): Our support team has created a set of Knowledge Base articles about how to make the change:

DNS Opt Out Servers
http://kb.earthlink.net/case.asp?article=187117

How to Specify DNS Information in Mac OS X
http://kb.earthlink.net/case.asp?article=173670

How to Specify DNS Information in Windows 2000
http://kb.earthlink.net/case.asp?article=187078

How to Specify DNS Information in Windows XP
http://kb.earthlink.net/case.asp?article=187109

Comments

Thank you.

I'm happy that you people have finally posted opt out servers for people who didnt already switch to other non earthlink DNS servers.

However, this is still unacceptable especially because you have the nerve to say you will be offering no customer service for issues with those servers. Goodbye earthlink.

NOTE: this comment follows your 'guidlines' yet somehow I doubt it will be posted like most of my comments.

As an Earthlink user, I appreciate at least having the option to use my browser's default handling of dead domains, but until this ridiculous "service" you've established is made opt-in (or removed entirely) I'll be telling all my coworkers, friends, and family to stay away from Earthlink. Breaking the internet in order to make money on advertising is a practice I'd rather not encourage.

I agree with the previous posters, everything has pretty much been covered already. I've been an earthlink subscriber for 5 years, and have gotten a number of people to switch to earthlink from AOL. That said, I will not continue as a customer if this is how Earthlink intends to behave and
l'll be looking into ISP alternatives if this behavior isn't fixed soon. It is a major annoyance, and disables a helpful feature (the browser automatically adding www. .com) for no good reason.

your statement:
we believe we are improving the EarthLink user experience with a system that will not interfere with other network processes.

If you've read the users' responses, you cannot possibly believe this. It's offensive for you to claim this in the face of the outrage you've provoked.

A previous poster said this, and I think it bears repeating:
Many people left AOL and joined Earthlink
because they resented AOL's interference in their Internet experience. Look
where AOL is now -- is this where you want Earthlink to go? This is a huge
step in the wrong direction.

This is OS hijacking and you have no right to do that. As much as you want to hide behind the RFC, you are only allowed to provide the users "RRs that the name server thinks will prove useful to the requester" and by no means spam. You're taking after AOL.
And yes, any change of the system configuration without the consent of the user is considered malware.

While it is nice that you included an option for people to opt-out of your forced DNS hijacking, it's a true shame that you consider these servers 'unsupported'.

This is what disconnects AOL from advanced and business users and you are heading down a slippery slope of repeating their same mistakes.

I have been patient so far hoping for a fix, but if this half-assed attempt at a workaround is all you can muster, considering your forced DNS changes BREAK standard operations, I am ashamed and will now work towards changing ISPs.

Adios EL. It's been a fine run but I will not put up with this garbage anymore.

While I appreciate that earthlink has finally responded to its customers, I still find it insulting that you characterize this dead-domain-handling as a "service" to your customers. It is a revenue-generating venture, and based on the overall level of arrogance displayed here by the marketing team, people's jobs appear to hang on the success of it.

Featured on InfoWorld / Ed Foster's Gripeline column today.

Unfortunately your solution is both too late and not acceptable. I've already moved my service to an alternate provider over this issue.

This just proves to show either:
A) How little your Engineers know about how DNS works
B) How stupid your PHB's are for shoving this down the Engineers throats.
C) A combination of A and B.

Just glad to say I stopped using EL a *long* time ago.

Not good enough.

Why should I have to jump through hoops, not be able to use DHCP to set up DNS, and deal with an unsupported EarthLink service just to get real Internet access? There are plenty of other providers that Just Work. So long, EarthLink.

A little late, but THANKS.

How nice it was that earthlink asked me if it was ok to break my browser and then tell me that fixing it is unsupported. I've setup a number of people on earthlink and all of tehm are now asking me why their "search from toolbar doesn't work". Now I have to take my time to go fix their DNS settings and use an "unsupported" configuration to do it.

If you think we are impressed by your graciousness in providing standard DNS servers, you are very sadly mistaken. I, like a number of other posters here will be looking for a new ISP.

IF IT AIN'T BROKE, DON'T "FIX" IT.

Clients *must* be allowed to determine the difference between a non-existent domain and valid one. With your DNS hack you have broken this - what happens when someone mis-types an email address and sends it? Are you collecting these too? What about anti-spam systems that work on invalid domain names? Why cant you people learn from other's (sitefinder) mistakes? Disable this selfish hack or face mass customer exodus to alternate providers who will provide stable DNS.

Over the past 20 years or so, I've watched the same ideas arrive, get stomped on, lose millions of dollars, then vanish, only to return again.

It's absolutely amazing that a company could watch another company make a huge blunder, then say "Hey! That looks like a great idea! Let's do that!"

Terry

1. I agree 100% with the above posters that EL's "east or west of the Mississippi, choose your preferred Earthlink DNS server" solution is pitiful and totally unacceptable.

2. I hope the executive(s) who approved this new but totally unethical "service" get(s) fired and can't find work anywhere in the industry ever again!

3. I am pleased to see the several pages of posts by the hundreds of people like me who are TOTALLY FED UP with big corporations trying to pull crap like this. I hope Earthlink goes under like the Titanic!

4. This morning, EL hijacked me AGAIN for about the 20th time in 2 weeks, even though last Thursday, I ran the google_rsearch.reg file to AGAIN (for the 3rd or 4th time in 2 weeks) to set Google as my default in-the-IE-address-bar search engine). So I decided I'm now angry enough to call our ISP (Embarq's DSL service) and ask them how to make this STOP!!!! None of the 4 DSL Tech Support Reps I talked with had a clue about this huge issue, even the "Network Engineer" to whom one of the Reps escalated me. Embarq doesn't offer any other "flavor" of ISP, so now this small business (a church) will be looking for another ISP and possibly also another local phone service provider. We will NOT be support something this insidiously immoral.

5. I'm going to do everything I can to call Earthlink's competitors (all sizes) and tell them to take full advantage of this window of opportunity for welcoming EL subscribers to the safe 'net haven they'll provide us.

I must agree with the other posters... The purpose of DHCP is to return IP address of your host as well as a DNS address. I do alot of traveling. Many times I'm not at home, so I want the local server to return the DNS address. If I hard code it as your "workaround" suggests, then I have to keep manually changing it every time I travel. This makes no sense at all. GET RID OF THIS NOW before you lose MORE CUSTOMERS.

THANK YOU

One more item... If you earthlink does this to your "data" service, can you image what they will do to your "voice" service? I definitely will NOT be going to Earthlink for Voice... Who knows what will happen if you dial the wrong number? Instead of getting the normal recording, you will be getting a bunch of ads as well as a dozen alternate phone numbers...

I was very upset at Earthlink when I could no longer use my computer the way "I" wanted to use it and had been using it for many years. I have been a long time user (started in the Mindspring days) but this has made me realize I need to become indepentant from any ISP. I will be moving to another service as soon as I make the necessary changes as I can no longer trust Earthlink.

Trust is important and once lost can never be regained.

This is not a solution. If you made it the default behavior and supported it, this would be a solution. As it stands, I will be sure to recommend that people avoid using your services.

You should check out the work that OpenDNS, http://www.opendns.com, is doing. They implement an "opt out" feature where a user can select not to get the alternate page when an Address can't be resolved. In addition they have a phishing filter that will automatically block known phishing sites... (also can be disabled). (Note, I don't use the Earthlink Toolbar... a topic for another Blog) If you AT LEAST IMPLEMENT an Opt-Out feature, similar to OpenDNS, then everyone will be HAPPY.
THANKS

This is just another example of how EL has just gotten too big to care. I was a loyal Mindspring, then EL customer for over ten years -- I kept a standard dial-up account for an email address. I also had a few hosting accounts set up for other companies. After six months of attempting to cancel one of the hosting accounts, repeated promises of "supervisor call-backs" that never came, I finally had my CC company stop processing the payments. When I got a notice of overdue payment for six months I called to cancel my last remaining account. It hurt to lose that email address, but not as much as it hurt being gouged and treated like I was an AOHell customer. Support your local ISP if you have one, and if not, find the cheapest business broadband account you can. None of the big guys give two craps about what the internet was founded to be. They all want you to only see their version of a proxied internet. No Thank You.

Wow, what a shame. I have always shunned value added isp type services since the advent of DSL, however, as a consultant I make recommendations to hundreds of clients...

Three guesses what my recommendations regarding EL will be, and the first two don't count.

I think this whole thing is absurd. Unless this horrible hijacking of my internet experience is removed in the next week I will cancel my Earthlink service. When I mis-type a URL or want to use my browser's search I should NOT be given a page full of ads, I already pay Earthlink enough money.

I plan to tell everyone I know to do the same. Offering the user a way to manually change their DNS is not a solution at all, it's a lame hack that only power users who don't travel or own laptops can use.

Thank you for providing this stopgap solution until I finish switching to a new ISP.

This is still too hard. There should be a gigantic "opt out" button the first time a user hits this page, or it should simply be ad free, or there should be a way for the user to share in the ad revenue.

It's actually easier for me to switch to a different ISP than it is to set the DNS by hand for the computers in our house.

Thanks for posting the "unsupported configuration", I suppose it is better than nothing. However, I still find the current default unacceptable and must ask earthlink as a paying customer to remove this new revenue maker. I do not see why earthlink should get away with making a buck off of me at my own expense.

Of all strategic intiatives that Earthlink as a company can undertake this is the best you can come up with? I would like to see your NPV for this project as your customers leave you for the competition. -

Here is another longtime customer that is saying goodbye!!!

I have been a new Earthlink customer for a few months since moving in with
new roommates. When DNS broke a couple weeks ago, I wondered what the
heck happened. Finally I tracked down the blog post about it. I'm glad
that you're having a "discussion" with your subscribers about this, and I
hope you realize it is time to put this harebrained experiment to rest.
There is a reason that customers pick Earthlink over AOL. By making this
petty money and eyeball grab, you are telling your customers you think
they are stupid and belong at AOL. Instead, this household will soon
switch to a service that offers INTERNET service with _working_ DNS,
unless this situation is resolved soon.

Also, I have been getting these Earthlink spam pages many times for VALID domain names. It is maddening.

Hi again Dave.

I'm pleased to see these new unadulterated Earthlink DNS servers which I and other customers asked for. Since use of these servers is currently unsupported, I assume this is just a temporary situation and that a more permanent and fully supported setup will be in place soon. If Earthlink has no plans to officially support the opt-out solution we asked for, I'll be forced to conclude that Earthlink is no longer a true ISP (like AOL) and will vote with my feet. An unsupported opt-out is not acceptable.

Followup to my previous comment--as many others have noted, this is not a solution. It's not even an opt-out, as if customers should have to figure out how to opt-out for working DNS. Opt-out would be if there was a website or phone number customers could call to make DHCP return working, supported DNS servers.

Two IP addresses for unsupported DNS servers is not going to cut it and is not going to quell the criticism.

I'm not an Earthlink customer, but am half-heartedly shopping around for a new ISP. I'm pretty happy with my ISP, but would pay a bit ($10/month) more for faster uploads.

I was considering Earthlink before I heard about this.

The rest of this post is somewhat off topic, but perhaps someone from your PR department will read this. Your marketing people keep mailing me advertisements for DSL. They don't include Earthlink's prices or speeds, but they do have a lot of inforation about AT&T's services.

I looked online to see what you offer, but you require way too much personal information to get lists of plans or price quotes. I don't have a land line, so I used a bogus phone number with a good area code. Your website said high speed internet is not available in my area...

Why don't you target your advertising more carefully, and/or publish your service plans and their prices online?

Too little, too late.

It's a shame, Earthlink used to be a real ISP. This pseudo-AOL behavior is disappointing at best.

Dave -
I would just like to comment that this DNS change severely impacted my abiltiy to work from my home office. I connect via VPN back to corporate, and I was not able to reliably connect to the various servers to peform my work. I am very disapointed in the fact that Customer Service could not address this ( I called three times, had the cable modem swapped, ran with a router, without, etc.) and let us know what was going on. I spent over two weeks self-diagnosing the problem and working with my companies IT group. Today, one of our techs finally figured out what was going on. After googling the symptoms, I found this blog. This simple solution you posted with the opt out DNS entries did the trick and took all of 30 seconds. What I am upset about is the two weeks I spent chasing this down. Please encourge Earthlink to properly address this issue.

Thanks

-John

Unsupported?

Nonsense.

I PAY you for both an internet connection and the support that goes along with that connection. If I choose to use these un-broken DNS servers and I have a problem with them, Earthlink WILL be providing me with support. It's just that simple.

You simply cannot legally offer support for some users and not for others.

This is a moot issue for me anyway, because I'm switching to a real ISP as soon as possible. Earthlink is not one anymore. I don't mean that in a reactionary sense, but rather a literal one - because you're not providing DNS service, you DO NOT provide unfettered internet access.

Fix the things you broke. Everyone hates this.

I too would like to echo the comments posted above. I figure if I keep silent, then Earthlink would believe I'm OK with this change.

For the past couple weeks, both my wife and I have been having trouble with our VPN connections. At first it was annoying and it only took me a very short time to figure out I was getting back garbage DNS entries for names of our Internal work computers... Fortunately I know how to work around the issue being that I work in the IT field...

However, that's not the case for most people. I can only imagine how frustrated a non-technical user would be when things begin to misbehave...

Therefore, I ask you, as a professional who has done more than a fair amount of end user support... Please, revert the functionality of your DNS server to return the appropriate responses.

Finally... I'm not threatening to leave simply over this one issue... However, this is just another reason, along with cheaper prices from my LEC for me to consider a switch... I have been a customer for over 6 years, however with these types of service changes, I won't be a customer for 6 more...

It is a shame to see that Earthlink has sacrificed our trust in order to make a profit by creating this ad portal "service". Failing to adhere to standards and common practices that are in place to allow the Internet to exist reveals a level of immaturity unconscionable, yet sadly precedented, for an ISP. You have lost the respect of Internet users by effectively making Earthlink DNS servers a large typo-squatting service. It is disappointing to see that Earthlink has not learned from the mistakes of AOL, VeriSign, and others, but has decided to inherit their stigmas.

This attempt at appeasement has failed to regain the trust of the technically adept community. It is an insult that underscores Earthlink's blatant disregard for the cooperative nature of the Internet and the responsibility that comes with being an Internet service provider.

For convenience, I have posted the list of "Core Values and Beliefs" from http://www.earthlink.net/about/cvb/ here. I urge Earthlink management to read this carefully and reevaluate your dead domain "service":

  • We respect the individual, and believe that individuals who are treated with respect and given responsibility respond by giving their best.
  • We require complete honesty and integrity in everything we do.
  • We make commitments with care, and then live up to them. In all things, we do what we say we are going to do.
  • Work is an important part of life, and it should be fun. Being a good businessperson does not mean being stuffy and boring.
  • We love to compete, and we believe that competition brings out the best in us.
  • We are frugal. We guard and conserve the company's resources with at least the same vigilance that we would use to guard and conserve our own personal resources.
  • We insist on giving our best effort in everything we undertake. Furthermore, we see a huge difference between "good mistakes" (best effort, bad result) and "bad mistakes" (sloppiness or lack of effort).
  • Clarity in understanding our mission, our goals, and what we expect from each other is critical to our success.
  • We are believers in the Golden Rule. In all our dealings we will strive to be friendly and courteous, as well as fair and compassionate.
  • We feel a sense of urgency on any matters related to our customers. We own problems and we are always responsive. We are customer-driven.

Wow! Two whole DNS servers for the entire country! If EL was serious it'd set up 2 in Columbus, two in Rochester, two in Poughkeepsie and so on. This system has zero redundancy. If you live east of the Mississippi and the single East Coast DNS server goes down, you're in a blackout. This shouldn't be acceptable to anyone.

This is a token gesture EL is making knowing full well that it's savvy customers are going to go elsewhere entirely for DNS services if not for all network services.

Alternative server considerers: Warning! This DNS is going to be slow! (When it's up, that is.)

Also, don't expect any "opt-out" functionality because such a feature would make the network overhead caused by EL's new rulesets to kludge this thing look like nothing in comparison. The guy who said this is going to result in unstable service is right. An opt-out would add to the fragility of EL's network.

All you guys who say you'll give it a week? Are you kidding? EL is committed to this; there is no going back. (Although these "unanticipated problems" Dave speaks of must have tossed their cost estimates out the window, so who knows?) Why wait another day? I'm already gone.

Preview of EL's next SEC filing:
Factors that may cause our revenue projections to fall short include wholesale defection of customer base and serious damage to our brand reputation caused by unbridled greed.


Kevin

I've been an Earthlink customer for 7+ years. This DNS-mangling is you persuading me to go away. If you don't put it back like it was pretty damn quick, your persuasion will work. Nuff said.

Mark has an excellent quote:

Perhaps Earthlink will end up in the same situation that AOL is in now and become a free, ad-drenched substandard ISP. Do you really want to compete with AOL in a race to the bottom or do you want to be a real ISP providing reasonable service? Is Earthlink's financial situation so bad that you have no choice?

I'm very unhappy and will now switch to RoadRunner Hi-Speed Online.

why didn't anybody learn from the verisign desaster and leave the DNS alone?
how much money does EL make with it, and how many man-hours are lost on the customer side?
i was an EL customer from 1997 till 2005 and saw the quality of the service going down - e.g. phone support and newsgroup quotas etc.

EarthLink is now indefinitely blacklisted from my list of potential ISPs to use. This move is a great disrespect to the structure of the Internet, and it is a great disrespect to those who use it. Relegation of true DNS behavior to the sidelines while staying with a default of "profit over protocol" is almost worse than having no true DNS behavior at all.

As just one example of an externality associated with this, imagine the extra support costs that will be offloaded onto other individuals and businesses who will have to cope with their users not understanding the origin of faults generated by this system. Providing real DNS as secondary and unsupported does approximately nothing to mitigate this.

I fully support and encourage any mass exodus from or boycotting of EarthLink that results from this decision.

Very disapointed and this unsupported DNS approach is unacceptable. Switching services tomorrow, case closed. I waited 23 days for a fix, Dave promised something and this is the best he can come up with? I'm through.

I been happy with my Earthlink cable for about a year now. About a week or two ago I started having problems with searching from the address bar. I thought my daughter downloaded some sort of malware again. Then I saw the Earthlink url in the address bar, which eventually led me to this blog. I am really disappointed that earthlink has decided to disable the proper DNS function and provide me with unwanted advertising in return (I'm just not seeing the upside to this hack...I mean upgrade, for the customer). If the two "opt out" DNS servers turn out to be too slow or Earthlink doesn't go back to the it's previous properly functioning DNS service, I guess I'll have to seriously consider going back to my old ISP.

The worst part about this is that these two DNS servers are not 100% unadulterated. Tested this on 2 broadband connections and found that sending DNS requests too quick can still end you back at the fun earthlink help page.

Or maybe I'm just an idiot and somehow missed changing the alternate dns... My bad.

I should point out that I spent three days (as I mentioned in my previous post) hacking my registry, not because I'm an idiot, but because the idea that this was all caused by EarthLink never even occurred to me. Why would it? That's how appalling this is. I only discovered this ginormous blunder once I began searching for information on the "bogus" search page in order to REPORT THEM TO EARTHLINK. "My ISP--EarthLink--will kick these fools where it hurts and teach them a lesson!" Boy, do I feel stupid. When my website was threatened with "hacking" by some idiot script kiddies, I laughed. First, they couldn't hack their way out of a paper sack. And second, even if they did, my ISP--EarthLink--would surely see them convicted for messing with their servers. Yeah! :(
Many have thanked you for this "open dialog". That's a crock. We only found this blog by accident and this is the only interaction we have had with EarthLink about this fiasco. Not exactly what I'd call an open and honest relationship with your customers.
Only a "noob" would consider this redirect to be helpful and shame on the tech guy for not setting up his wife's browser any better than it apparently was. Much like the ads we receive via e-mail--spam--I must inform anyone who doesn't already know better never to click on any of the links. When you want V**gra, do a search and find it. Every time you participate by clicking on a link, YOU PAY TO BE SPAMMED--or redirected in this case. Pass it on. This "search" page is now blocked until I switch ISPs.
This has also caused my SpamBlocker to catch e-mail from senders who are already in my EarthLink Address Book to be caught as spam. I have re-added these addresses several times now--to no avail.
Unfortunately, we who regard this all as blatantly foolish are in the extreme minority. The vast majority of PC users are morons--as they are in every other area of life these days. Welcome to "Earf" and the decay of society, in case you weren't paying attention. What that means is that this will continue, no matter how many of us leave EarthLink, because we don't matter to this new world of no education, no morals and no hope. Children are the future--and the future is dim. Yo. Ask any 20-something with a generic tribal tattoo who thinks cheat codes are the way the game is played. Ask any obese 12-year-old what hard work and values get you. The answer will probably be a B*g M*c and a blank stare.
The lady on the EarthLink commercial says, "I will do everything in my power to make your problem my problem..." That always cracks me up, since it IS YOUR PROBLEM every time I have an issue. Hahahahahaha! Every time I have contacted EarthLink Support, it was to inform them that their server was compromised or something of that nature. Which is why I didn't bother with that before I spent three days trying to figure out how and who had hijacked my browser. And it wouldn't have mattered anyway, since the techs weren't even informed about this. So funny now. :S
By the way, as you can clearly see, I DO KNOW HOW TO SPELL. Look, ma, no spell check!
This isn't about BS "fixes" or ads. This is about being ashamed to be associated with you--EarthLink. This is actually EMBARRASSING for some of us who have talked all of this junk to our pals about our SUPERIOR ISP--EarthLink--and laughed at those who used AOhelL. But whatever, mannnn. My head is numb.
I *HEART* Tiffany Flipplocks (even though I'm fairly sure she's actually a he).
*Just approve this for the comedic value. It's the least you can do for me. Thanks in advance.*

Note that EarthLink folks Dave and Ken Womack keep referring to this as the "dead domain system." Note the lack of hyphen, which would have indicate a compound adjective describing the system:

Dead-domain system = a system that handles dead domains

Dead domain system = a domain system that is dead.

I think EarthLink is being more accurate than they intended.

FYI: Here are very clear and detailed instructions for changing your DNS servers from OpenDNS.com. There are instructions for routers and single computers running various operating systems.

To use Earthlink's unadulterated DNS servers instead of the OpenDNS DNS servers substitute IP addresses "207.69.188.171" and "207.69.188.172" where the instructions mention OpenDNS servers "208.67.222.222" and "208.67.220.220". Note that "207.69.188.171" comes first if you are on the west coast, and "207.69.188.172" comes first if you are on the east coast. (if you are somewhere in between I guess it's a toss-up, although you could use "ping" to see which one responds faster).

I hope this helps some people. Posting this information does not mean I condone what Earthlink has done, I think they should scrap the bogus DNS system (aka "Dead-domain system") right away if they can't make it OPT-IN.

Hi,

My earlier comment was censored.

The thing is the pointing to an alternate DNS server is not an option with my router. So the solution above does not work for me.

I have switched to RoadRunner already. I have no patience for this. Thats what you should: switch, and you'll also get promotional rates from the new service provider :)

In any case, I've put a link to this fiasco on my blog.

Ooops, sorry, here it is. I made a mistake in the earlier links by linking to blogpsot instead of blogspot.

http://chillifrost.blogspot.com/2006/09/canceling-earthlinks-internet-service.html

This is not the right response. I'm sure it's done with the best possible motives, but Earthlink, as one of the few ISPs left that takes the Internet seriously, needs to rethink this entire strategy.


Remember Web browsers are perfectly capable of handling the issue you're trying to "fix", and pretty much all of them do - only according to the user's preferences, language, etc.


Once my contract is up, I may seriously have to look again at ISPs like Speakeasy, which previously I've rejected due to a lack of apparent competence, but Earthlink's not showing itself to be any better, indeed on a key issue, it's showing itself to be the opposite.

Hi, If you are reading this, you probably have been relatively happy with Earthlink. And the reason why you are venting is that you want Earthlink to fix it. I feel the same way. I have been relatively happy with Earthlink and don't really want to change unless I'm forced to (and this will force me to)

Here is what I'm doing to put the pressure on Earthlink...
I'm going to file support requests complaining about this and try to post information about this in other news organization. I hope between the internal pressure (from support) and external pressure (from the news media) Earthlink will come to their senses and remove this dumb feature.

GOOD LUCK

I'm happy to say - the Verizon FIOS installer will be here next week. ByeBye Earthlink, I hope your entire company sinks into the depths of hell for breaking fundimental rules like this.

CAUTION TO ALL OF YOU: The servers earthlink is giving out for you to use will be SLOW AND UNSUPPORTED. DO NOT USE EARTHLINK SERVERS EVER as they are no longer acceptable.

Instead use something like Verizon's servers. Try: 4.2.2.2 and 4.2.2.1 works for me faster than Earthlink ever did. Getting FIOS in a week anyway so I dont have to put up with any future *IMPROVEMENTS* from earthlink.

Another article, from "Publish": How Earthlink Broke the Internet.

I had already switched to non-Earthlink domain servers. Until these problems began and I started searching for solutions, I didn't even know I could do that. I would have appreciated being informed about Earthlink's page pushing beforehand, rather than being forced to troubleshoot the interruption in service on my own. Earthlink tech support even seemed to be left in the dark as they assured me Earthlink had made no changes. In a matter of days, I've forgotten years of trust. There's still a void of information. I'd like to know if Earthlink is considering trying to return to what it was or is committed to its new character. I decided this month not to entrust my voice communications to Earthlink. I'd simply like enough information about Earthlink's path to decide whether I'll be on it.

Well, at least there's a workaround. You've managed to break the really nice feature in Mac web browsers where you only have to type 'domain' instead of 'www.domain.com'.

What I don't understand is why you didn't put this functionality in a toolbar that can intercept nxdomain responses and query your earthlink-help page so only noobs would have it and everyone else could opt out. That is the only acceptable answer.

I happened to read about this yesterday, and today found out that a secondary DNS server copied from an home connection and put in our spam firewall at work to help alleviate timeouts when our isp was having some service issues caused many legitimate emails to be blocked and attributed to a DNS blacklist they were not listed on. When I finally found the DNS servers and saw an Earthlink one I knew exactly what the problem was. I quickly corrected that problem, and when I get home I will begin calling the cable company to switch to Road Runner.

Great. I tried to watch a video on my PVR system last night and couldn't, because this "helpful" service had decided to start returning incorrect info for names that are not domains. Rather than properly mount my samba share to access the video, it tried to connect to an IP that I have nothing to do with. I believe that the querey for \\computername somehow was returned with the ip of computername.net or .org (can't remember the exact address, but it didn't resemble the earthlink-help .net ones. I don't want domain servers that try to guess what I want (not only did they get it horribly wrong in this case, computername.com, .net, .org, .co.uk, etc. etc. all exist - why the heck would you ever try to return one of those when asked for something that could be any of them?

Too little too late. On our EL connection, most of us have laptops, which we use at home and elsewhere too. We depend on DHCP at each location. Our DSL modem from EL does not allow us to permanently change the DNS servers--they are reset to the default after a power cycle.

I just signed up for Qwest DSL. I gave you guys a couple weeks to see if you'd come to your senses. I can't afford to waste any more time while you guys count your money.

The sad thing about all this is that I was satisfied with the service until this DNS business. You broke it. No matter how much you or Barefruit insists it's a feature, you broke it.

Either your network administrators should be fired for allowing this move to go forward unchecked, or your board of directors should be fired for the naivete or bullheadedness to foist an ill-advised venture like this on your *entire* customer base.

How could you possibly screw this up? I mean, all I ever wanted was raw Internet connectivity. That's it! If I wanted an ISP that tries to do all my thinking for me, I'd go to AOL.

Apparently Earthlink is the *new* AOL. Your core competency is no longer network connectivity, it is milking your customers dry.

Good bye and good riddance.

Comments on this entry: 63 and counting.

Comments on your other entries: 0 and holding.

THIS is the issue your readers care about. Solve the problems.

Matt - The Samba mount problem sounds like it could be related to the VPN issue mentioned in the blog entry above -- if you aren't already you could use the DNS server settings suggested above that are outside of the standard DNS system, and that should steer you clear of the problem.

Dave C. wrote:

Matt - The Samba mount problem sounds like it could be related to the VPN issue mentioned in the blog entry above -- if you aren't already you could use the DNS server settings suggested above that are outside of the standard DNS system, and that should steer you clear of the problem.

Maybe he could, but why should he have to? For some people it's not even possible to override the DNS settings returned by DHCP, and for everyone else it's an inconvenient hack. Earthlink should provide standards compliant DNS by default. Your marketing tail is wagging your technical dog.

In an earlier post you said:


... We'll be watching the system closely and analyzing performance data to make sure we're right about this, and we're listening to our users as well as the conversations online.

You many be listening, but apparently it doesn't matter to you. Users are saying you are wrong about this and should go back to returning valid DNS results.

How about an update, Dave?

You've been awfully quiet about this for a few days now, and it's still an enormous problem for everyone.

What's the story?

Why is Earthlink ignoring its customers?

It's clear by the comment volume that the "dead domain handling" threads get that it's a *BIG* issue to many people.

Where's the Senior Project Manager now?

I can't believe you have left this to fester this long. I signed up for Bellsouth DSL today, and I'll be dropping Earthlink just as soon as I get all my email correspondents notified.

Today I have experienced losing my internet connectivity twice. Both times this happened I was able to restore connectivity by removing the alternate DNS' provided in this post and allowing for Earthlink's default DNS to take over.

While I understand that alternate DNS' are a "self service solution," albeit to an unnecessary and unwanted problem, I would think that such a remedy should work consistently.

I can't believe that the silence of unknowing customers, oblivious to change, somehow outweighs the dissatisfaction of other paying customers with a bit more savvy.

I'm just amazed that this "fix" for bad domains was put in place. This is only a solution if your marketing and engineering teams honestly belive that the internet is for browsing only (and they never read the RFC's to see how the thing was supposed to work).

How long until you remove those servers?

And to think: this was the "good idea" that came out of a series of internal meetings.

The mind boggles at the brilliance...

I am very glad you have a bypass so we can use DNS as it was designed rather than this drivel you have released.

These URLs don't work for me either. How about a more viable solution?

I contacted Conumser Reports to make them aware of this issue and they said that they had already been notified, but that if more consumers that contact them, there is a better chance that issue will be published in their magazine. I suggest that others users conact Consumer Reports as well!

Say, everybody -- don't we all think Dave C. merits some credit for even working this thread with us? Some of what people are posting here is basically shooting the messanger, which is, I think, dumb. I don't know Dave, he's probably just a decent guy who's trying to get through his day in one piece like the rest of us are.

Anyway, I'm giving up here, just signed on with TW cable, good luck to everybody.

To those of you who have asked for an update or are concerned I've moved on -- I understand the confusion, especially if you're new to Earthling. My beat is pretty broad, and it's my job to publish new stuff every day, most of which is different from what I published the day before. On this issue, things stand where you see them now, but just like everything I handle on the blog, I'll continue to cover any new information as I hear it, and your comments and emails are always read and considered as long as they follow my ground rules for discussion.

leo - I'd like more information. I do know that the DNS addresses above are functioning fine -- please e-mail me if you'd like instructions for how to use them.

Ok, I tried to post about this before...but apparently my post wasn't "approved." I keep trying to search using the address bar of IE6, which worked absolutely FINE before this so-called improvement showed up, but now, it keeps hijacking my searches and telling me that the website is not found. Well, duh, I'm not entering a website url, I'm performing a search, and I want search RESULTS, not a bunch of random links at the bottom of a page that tell me absolutely nothing I want to know.

Thanks for the update Dave.

Two important things you *didn't* address:

1) Is earthlink senior management aware of the volume of compaints about this?

2) Are the "opt-out" servers supported at the same level that the other, marketing-enabled servers are?

Sheena: there are a couple of things you can do if you want to be able to search that way from within IE6. If you use the EarthLink Toolbar, there's a Google search interface built in at the top. Or if you want a different search interface or prefer the way IE6 works, set your computer's configuration to use one of the two alternate DNS servers we've listed above.

Joe: on 1., Many employees read Earthling themselves, and when there's a topic that gets a lot of discussion I also make sure that the project teams and leadership are made aware of it.
On 2. The alternate DNS servers listed above are supported by our systems administration staff for configuration, monitoring, and service level, like our standard DNS servers.

Dave C said...
[[[
...if you aren't already you could use the DNS server settings suggested above that are outside of the standard DNS system, and that should steer you clear of the problem.
]]]

Personally, I consider these "alternate" servers to be PART OF THE STANDARD DNS SYSTEM.

It's the default EL DNS servers that are "...outside of the standard DNS system...".

Like others, I find this kludge to be a hack, I have to keep changing my IP settings each time I move my laptop between work and home; this is unacceptable. Please make the DNS behavior an option selectable on the EL customer account.

I just wanted to quote an insightful comment on slashdot by Keith Moore, a CD Professor at University of Tennessee:

The biggest problem with this is not the ads (though they are annoying). This DNS hack doesn't just affect HTTP, it affects every application that does DNS queries. The claim that the system is configured to only handle NXDOMAIN HTTP traffic is a bald lie. There is no way for the DNS server to determine whether a query is being done for HTTP or for some other protocol.

When an application queries DNS for A records (IPv4 addresses) for a particular domain, one of three things should happen:

  1. If there are A records for that domain, they should be returned

  2. If there are no A records for that domain but there are other records, "no information" should be returned

  3. if there are no records of any type for that domain, "no such domain" (NXDOMAIN) should be returned

What Earthlink's servers appear to be doing is the following:

  1. If there are real A records for that domain, they are returned

  2. If there are no A records for that domain, return A records for several hosts that don't belong to that domain. if the application tries to talk HTTP to port 80 on any of those hosts and supplies the Host: query request (standard in HTTP 1.1) the HTTP server will do a search for the domain that appears in the Host: request and return HTML that suggests other domains that appear to be similar to the one given in the Host: request. however if the application tries to talk to other ports on that machine it will get "connection refused" or it will time out.

(the behavior is actually a bit more complicated than that. the behavior seems to be dependent on the IP address from which the queries were made - so if you make the query to their servers from a host that isn't on Earthlink DSLyou will apparently get normal results. the behavior also seems to be dependent on the domain being queried.)

There are several things wrong with this behavior:

  1. It's not reporting the error correctly. Applications that do DNS queries quite reasonably expect NXDOMAIN to be returned if the domain does not exist, and "no information" to be returned if there are no records of the type they're looking for - not a list of apparently valid IP addresses pointing to hosts that have nothing to do with that domain. Many applications behave differently depending on the error condition. "connection refused" and "connection timed out" are often treated as temporary errors - the application assumes that the remote server is rebooting or isn't reachable and tries again later. "no such domain" is more often treated as a permanent error, or one that requires immediate user attention. So this Earthlink change can cause applications other than web browsers to behave improperly, or to give misleading error messages.

    For example: if an email server is trying to send mail to someone at a particular domain, it will first do a query for MX records to determine if there are any mail servers assigned to that domain. If the MX query returns no answers, it may then issue a separate query for A records. If this happens the Earthlink DNS server will return bogus A records and the email server will try to send the mail to Earthlink's servers rather than bouncing the mail like it should. When Earthlink's servers refuse the connection, the email server will treat the condition as a temporary error and retry at intervals for several days. As a result, mail for nonexistent domains (say, bounced spam) can clog up the email server's queues and slow things down.

  2. It is hiding other records associated with that domain. Say an application will talk either IPv4 or IPv6, but prefers to talk IPv4 (which is reasonable because these days IPv4 is often faster than IPv6). So the application does queries for IPv4 addresses ("A records"), and if it finds none, does queries for IPv6 addresses ("AAAA records"). But if there are no A records for that domain Earthlink's servers appear to return the bogus A records pointing to their own servers. The application will try to contact those records and fail. It might never try doing the query for the AAAA records.

  3. It adds additional delay. Even if the application doing the DNS query treats "connection timed out" or "connection refused" essentially the same as "no such domain", "connection timed out" takes a lot longer, and "connection refused" takes slightly longer - especially if the application tries to contact each of the bogus servers before giving up. So while this change might improve service for users of web browsers, it drastically reduces the level of service for other applications.

  4. Even if the application uses port 80, that doesn't mean it's using HTTP. Even if the application is using HTTP on port 80, that doesn't mean it's expecting ordinary web pages. For better or worse many applications use port 80 to get through firewalls, and many applications use HTTP to obtain files other than human-readable web pages. Earthlink's DNS hack will confuse applications that use port 80 and/or HTTP by responding in ways not appropriate to that application.

  5. It misrepresents the content of the parent zone. Every domain name is contained within some parent domain. A zone is just a list of records associated with a parent domain. For instance, foo.example.com is contained within the example.com zone. The list of records under example.com is owned by whoever currently owns the example.com domain. For Earthlink's servers to claim that foo.example.com has A records when it doesn't have A records is to lie about the contents of the example.com zone.

I admit to being a bit confused. Are you saying that Ken Womack
(Senior Product Manager, Search) is not correct when he said, "...but this isn't something that you'll be able to get customer support help with."? Are these or aren't these supported?

Also, what is being done in terms of compensation for your customers that were out of service for weeks, until this issue is resolved? Since this is very clearly something that was caused by EL, it seems appropriate that EL takes responsibility for it.

I don't understand why the Sept. 25 issue of e-link did not mention this new DNS "service."

I tried to ask the e-link people using the e-link form at http://www.earthlink.net/elink/email/eform.html but the form just returns an error message.

The DNS mangler now employed by EL has bitten me now too. Doing local network machine name lookups before the browser master lists were syncronized caused me quite a bit of confusion. After pinging a local machine and noting a non-local IP returing the replies I figured my laptop must have been hijacked by some malware. I would never have thought it would have been hijacked by my own ISP.

I'e been considering switching providers for some time now, since I don't even need the required phone line which adds $30 to the cost each month. I think this was the straw that broke the camels back.

Whoever pushed this through at EL should be fired.

Clarification:

This isn't what we meant by "opt out". Overlooking the fact that you selected the least acceptable of what most of us considered acceptable solutions, you've botched it.

If it MUST be opt-out, it has to be a setting on our EarthLink account, which instructs EarthLink's DHCP servers to point us to working DNS servers.

Expecting people to manually enter DNS information is simply not going to fly. Too many other ISPs do it correctly.

this is horrible...I am leaving EL.

Just curious: I was using the 171/172 DNS servers to get around the bogus-DNS issues from EL. But starting last night (26Sep) even they stopped working.
I ended up routing my DNS thru some external non-EL servers instead.
Did anybody else start having problems with the *normal* DNS servers they told us about?

G

DSR: Yes, to clarify, the alternate servers are being supported as far as server configuration, monitoring, and service level. As to your second question, I'm not aware of any customers being out of service as a result of this system, but if you've experienced any service outages that's an individual question for customer service.

BillT: I'll report the problem with that form to the ELink folks.

Garrett: Thanks for passing this along. I don't do the sysadmin/support work on the servers myself so I don't know off-hand. I'll pass your comment along to our engineers.

I have posted a story to slashdot on this to see what that group thinks. Over the years, earthlink is trying everything they can to make money via "value added" services. I guess this is a marketing thing they are expecting to have fly right by the customer base. For those of us that have stuck around out of loyalty and laziness for the last 10-12 years, it might be time to move to a telco or cable provider since i have seen no proactive improvement to service from earthlink other than clogging the pipe with more useless bandwidth of ads. Sorry for the rant.

I am in no way a DNS geek (or whatever they called it), an engineer, an IT admin or anything else like that. So i hope EL will not marginalize my comments as it seems to have done to those others.

That said, i have been an EL DSL customer for about 5 generally happy years. That changed when my search from the address line function changed without any notice or warning to me; nor any desire on my part that it do so.

After clearing cache and cookies and temp folders and all of that-- which was a major inconvenience since i then had to re-enter all my id's and passwords-- i finally called EL.

The assistance was slow and confused, and then when they finally provided me this work-around, i still had to ask why it changed in order to be sent to this site for more info.

I was never told the server would not be supported. It seems that would be a good detail to provide to customers. Anyway-- i wish this wasnt an EL site because i want recommendations from the experts EL seems to dismiss about which ISP i should seek out as a replacement.

Granted it wil probably take me a while to switch, and i really hope EL is not so short-sighted that it will not re-think (or actually think about it for the first time) this decision to proceed in this manner. I really would prefer to keep my email address. But there is no question in my mind that i will give it up if this is what i can expect.

Hopefully, i can find that info when i search from the address line.

You ruined a pretty good think EL. You still have time to make it right. And hopefully, to learn from this debacle.

Geez!

jackky:

Slashdot has already done 2 stories on the topic:

EarthLink Establishes Their Own "Site Finder" (9/3/06)
and
Earthlink Offers Alternate DNS Without "Dead DNS" (9/18/06).

See also: InfoWorld / Ed Foster's Gripeline column 9/18/06 and
"Publish" on How Earthlink Broke the Internet.

A Google news search on "Earthlink DNS" will show some others (sort by date).

I was until yesterday, a longtime 12 year customer of Earthlink. My speeds have decreased over the last four months. I finally get Roland a senior tech to authorise a replacement modem without having to buy one and sign new one year contract. The modem never shows up. The support people in india have no record of the call or what was promised, even though i give them the ticket number. I ask to be escalated to a senior tech, then then agrees to what Roland authorised. All this time I'm using a neighbours WiFi access to get online. The modem arrives on friday, All is well until Sunday when this Dead Domain thing shows up. I spend nearly 4 hrs between monday and tuesday telling your India tech support people what is happening to me. They keep denying that such a thing is happening. Frustrated beyond compare I give up and tell them thanks and hang up. Find this blog on Weds morning and i have since cancelled my account with Earthlink. A shame 'cause you have the best spam filtering available.
I am now a customer of my local cable company now.
I am also the only Apple Computer consutant and tech guy within a 150 mile area and have referred everyone of my clients to Earthlink, many of them people who had left AOL because of AOL's constant in your face instrusion . After this effort by Earthlink for a new revenue stream from advertising I will no longer be able to refer my clients. When they ask for an isp, I will tell them to look at the cable provider I use and to stay clear of Earthlink

I'm still a bit confused. You wrote: "I'm not aware of any customers being out of service as a result of this system"; after reading the three different blog pages on this topic, how can you say that nobody was suffering service issues? If the service isn't functioning properly, I would call that "out of service"!

As for speaking to customer service, they keep insisting that I have to talk to Time Warner Cable about some sort of discount, because they handle the billing. It seems kind of odd that TWC should have to provide a discount on something that was entirely earthlink's fault.

Hi dave,

I don't think you have answered this question. If you have, I apologize for asking again.

How do I get these normal DNS servers to get loaded automatically instead of the messed up ones? I don't know that much about computers and I don't like to have to fool around with them. Can you post a tutorial or something?

I hope there's an answer because I'm pretty PO'd that Earthlink has messed up my online experience. I like to use the address bar to search like it is meant to do. I feel its not unreasonable to expect the ISP to leave me alone.

Thank you in advance.

Bruno

DSR: The technical issues we've been seeing (and working to resolve) are things like the ability to use your location bar as a web search mechanism, and VPN issues. Although I don't want to minimize inconvenience this may have caused you, I don't know of customers who have been out of service because of it. I can't speak for all customers, but I do know that for the most part the single word search stuff appears to be functioning fine, as is our goal. As far as your individual concerns about your experiences, it sounds like you're in touch with the right people and it's something you can follow through with them on. I can help get you through to the right group, but beyond that it's really up to you, them, and the details of your case.

Bruno: Our support team has prepared a set of Knowledge Base articles that explain how to make the change:

DNS Opt Out Servers
http://kb.earthlink.net/case.asp?article=187117

How to Specify DNS Information in Mac OS X
http://kb.earthlink.net/case.asp?article=173670

How to Specify DNS Information in Windows 2000
http://kb.earthlink.net/case.asp?article=187078

How to Specify DNS Information in Windows XP
http://kb.earthlink.net/case.asp?article=187109

There is a slight change to the wording in the first entry that will be made momentarily, but in case you see it before then, here's how it should read:

"Customers east of the Mississippi should set the East Coast server as the primary or preferred DNS server and the West Coast server as the secondary or alternate DNS server. Customers west of the Mississippi should set the West Coast server as the primary or preferred DNS server and the East Coast server as the second