02 Novembre 2009 Changes (HHH) ------------------------------ 1. Action: ntpagetag Added: BadURL_Parts[i++] = "ntpagetag"; // Tracker - 2009-10-05 Reason: Was used at epiduo.com. The script looked AWFUL. They did their best to hide EVERYTHING. 2. Action: alexametrics.com From: BadDomains[i++] = ".alexametrics.com"; // PRIVUS Tracker - 2009-08-29 // PRIVUS Traqueur - 2009-08-29 To: BadDomains[i++] = ".alexametrics.com"; // Tracker - 2009-10-05 // Traqueur - 2009-10-05 Reason: I have had it for over a month and all because I saw a host in this domain named atrk.alexametrics.com that I used to not block (RTS). Now I do but this rule also is now in force for everybody. 3. Action: added another exclusion for YouTube. Added: GoodDomains[i++] = "youtube-nocookie.com"; // 2009-10-05 Reason: in my logs and is is supposed to be an opt-in for tracking at YouTube. Supposedly, it didn't work. You got the Flash cookie anyway. Do you get it now? All I know is that when I did a wget on the URLs in my phttpd log, all I got were tracking 1x1 GIF images. But since it is just YouTube with a twist I have to white-list it. 4. Action: Added some personal rules Added: BadHostWordStarts[i++] = "adserver"; // PRIVUS AdServer - 2009-10-06 BadHostWordStarts[i++] = "click"; // PRIVUS Tracker - 2009-10-06 BadDomains[i++] = ".firstlightera.com"; // PRIVUS - AdServer - 2009-10-06 Reason: adserver is being tested to put in for others. click will probably be forever PRIVUS. It is just to be used to help me capture clickers for hosts file inclusion. It is too early to tell what I am going to do with firstlightera.com. I just want to know what it is. At first I thought it was an ad server but now it is beginning to look like a secondary content delivery system. But then there is this: http://www.killerstartups.com/Site-Reviews/\ firstlightera-com-editorial-related-advertising { 2010-01-23: First was added, 2nd has too many false positives and is removed. NOW YOU KNOW THE VALUE OF TESTING A PRIVUS RULE - KILL YOURSELF, NOT SOMEBODY ELSE! 3rd IS a secondary content system. This rule would NEVER have been give to others. I got tired of adding it for myself since it never stopped anyting until the very end of the 2009 year. } 5. Action: Did some formatting for diff. Date: 2009-10-07 From: BadHostParts[i++] = "celeb"; BadHostParts[i++] = "codec"; // VOTRE CHOIX BadHostParts[i++] = "oasc"; BadHostParts[i++] = "valueclick"; To: BadHostParts[i++] = "celeb"; // Malware - 2008-12-07 BadHostParts[i++] = "codec"; // VOTRE CHOIX - TROJAN BadHostParts[i++] = "oasc"; // AdServer - 2009-10-07 BadHostParts[i++] = "oasc"; // AdServeur - 2009-10-07 BadHostParts[i++] = "valueclick"; // AdServer - 2009-10-07 BadHostParts[i++] = "valueclick"; // AdServeur - 2009-10-07 Reason: So things that can be the same ARE the same. NOTE. There are some unresolvable differences: -------------------------------------------------------- BadURL_Parts[i++] = "adult"; - in pornproxy* BadHostParts[i++] = "adult"; - in proxy* -------------------------------------------------------- BadURL_WordStarts[i++] = "girl"; - in pornproxy* BadHostWordStarts[i++] = "girl"; - in proxy* -------------------------------------------------------- BadURL_WordEnds[i++] = "girl"; - in pornproxy* BadHostWordEnds[i++] = "girl"; - in proxy* -------------------------------------------------------- BadURL_WordEnds[i++] = "girls"; - in pornproxy* BadHostWordEnds[i++] = "girls"; - in proxy* -------------------------------------------------------- BadURL_WordStarts[i++] = "sex"; - in pornproxy* BadHostWordStarts[i++] = "sex"; - in proxy* -------------------------------------------------------- BadURL_WordEnds[i++] = "sex"; - in pornproxy* BadHostWordEnds[i++] = "sex"; - in proxy* -------------------------------------------------------- Some are like "nude" and "adult" that show up as diffs but they are IDENTICAL in both files. 6. Action: Moved "celeb" rule where it was supposed to be. Date: 2009-10-09 From: URL_Parts To: HostParts Reason: IT WAS TOTALLY WRONG IN THE pornproxy* files! 7. Action: Activated tracking rules for EVERYBODY. Date: 2009-10-09 From: BadURL_WordStarts[i++] = "wtbase\.js"; // PRIVUS Tracker - 2009-09-19 BadURL_WordStarts[i++] = "wtid\.js"; // PRIVUS Tracker - 2009-09-19 BadURL_WordStarts[i++] = "wtinit\.js"; // PRIVUS Tracker - 2009-09-19 To: BadURL_WordStarts[i++] = "wtbase\.js"; // Tracker - 2009-09-19 BadURL_WordStarts[i++] = "wtid\.js"; // Tracker - 2009-09-19 BadURL_WordStarts[i++] = "wtinit\.js"; // Tracker - 2009-09-19 Reason: No false positives. I still cannot release this one that logically is related to them but it will be if it works well. BadURL_WordStarts[i++] = "dcs\.gif"; // PRIVUS Tracker - 2009-09-19 { 2010-01-23: It is public now. } 8. Action: Added some SPAM IP rules Added: BadNetworks[i++] = "60.12.166.154, 255.255.255.255"; // SPAM - 2009-10-10 Reason: It was the most current one of these spammers at Google. Just don't expect it to hang around. They will be gone in just a few months, Maybe just one or two months. 9. Action: Moved all the REGEXPs in the BadHostWordStarts to the END and reordered everything. Date: 2009-10-11 Reason: To make diffs work better and also to make the point where the REGEXPs should be placed clearer. 10. Action: BadHostWordStarts[i++] = "girl"; Date: 2009-10-11 Added: BadHostWordStarts[i++] = "girl"; (BUT ONLY TO THE PORNPROXY* FILES Reason: This rule was downgraded from URL to Host status in ONLY the proxy* files but was left as URL in the pornproxy* files. This is making it a little clearer for diff so it just looks like a rule was removed. 11. Action: Moved all the REGEXPs in the BadURL_WordStarts to the END and reordered everything. Date: 2009-10-11 Reason: To make diffs work better and also to make the point where the REGEXPs should be placed clearer. 12. Action: BadHostWordStarts[i++] = "sex"; Date: 2009-10-11 Added: BadHostWordStarts[i++] = "sex"; Reason: This rule was downgraded from URL to Host status in ONLY the proxy* files but was left as URL in the pornproxy* files. This is making it a little clearer for diff so it just looks like a rule was removed. 13. Action: Moved all the REGEXPs in the BadHostWordEnds to the START and reordered everything. Date: 2009-10-11 Reason: To make diffs work better and also to make the point where the REGEXPs should be placed clearer. 14. Action: BadHostWordEnds rules added to pornproxy* files Date: 2009-10-11 Added: BadHostWordEnds[i++] = "girl"; BadHostWordEnds[i++] = "girls"; Reason: These patterns exist as URLs only in the proxy* files since these rules were downgraded to HOST status for those files but are left as URL in the pornproxy* files. BY doing this, the diff is easier to look at. 15. Action: Moved all the REGEXPs in the BadURL_WordEnds to the START and reorganized everything. Date: 2009-10-12 Reason: To make diffs work better and also to make the point where the REGEXPs should be placed clearer. 16. Action: Added a redundant rule for diffs of pornproxy & proxy Date: 2009-10-12 Added: BadHostWordEnds[i++] = "sex"; Reason: The regular proxy file downgraded this rule from URL status to just HOST status. I have added this new rule to the pornproxy* files to make it easier to read the diffs. 17. Action: Added redundant BadHostParts rules for diffs of pornproxy and proxy files Date: 2009-10-12 Added: BadHostParts[i++] = "adult"; BadHostParts[i++] = "[^hn]cock"; BadHostParts[i++] = "gay"; BadHostParts[i++] = "huge"; Reason: These rules were downgraded from URL to HOST level in the proxy* files. I am adding them back in at this lower level from the proxy* files into the pornproxy* files to minimize the differences in the diff utility. BadURL_Parts[i++] = "virgin[^im]"; 18. Action: Added a spam rule Added: BadNetworks[i++] = "218.10.16.155, 255.255.255.255"; // SPAM - 2009-10-12 Reason: A high enough of a representation at my GMail account and the fact that unlike the other one they have links causes me to ADD THEM. 19. Action: Tracker rule Added: BadURL_Parts[i++] = "piwik\.js"; // Tracker - 2009-10-13 Reason: WHY NOT? It tracks, I redirect the request to myself and that is the end of that. 20. Action: Removed the BadHostParts rules I added in 17 for Rodney's diffs. NOW HE IS COMPLAINING THEY ARE THERE! SHEESH! There is one that will change though. I am demoting the "gay" rule to Host status in the pornproxy files as well as the proxy* files. That is because we also have start and end rules. So that is a removal from only the Date: 2009-10-19 Removed: BadHostParts[i++] = "adult"; BadHostParts[i++] = "[^hn]cock"; BadURL_Parts[i++] = "gay"; BadHostParts[i++] = "huge"; Reason: Rodney is just going to have to live with the diffs. Next he is going to to have to live with the removal of the rules we added in 10, 12, 14, & 16. 21. Action: New Omniture site-specific code Added: BadURL_WordStarts[i++] = "s_code\.js"; // PRIVUS Tracker - 2009-10-20 Reason: Saw at DeanVariety.com which means it is used by GuthyRenker. Others may or may not use it and it is NOT in the ABP EasyPrivacy subscription. The rules for omniture_code.js and s_code_remote.js are in 35. { 2010-01-23: It is public now. } 22. Action: New tracker services I have added manually Added: BadDomains[i++] = ".clicktale.net"; // PRIVUS Tracker - 2009-09-28 BadURL_Parts[i++] = "ads\.js"; // PRIVUS Tracker - 2009-08-30 Reason: I got tired of adding them manually. The second one is actually too short but I had problems with the full anchor pattern so I am trying to find what false positives I get with this one. { 2010-01-23: Both rules are now public. See next . } 23. Action: They have been private long enough and ABP's EasyPrivacy blocks some of these carte-blanche anyway. From: BadDomains[i++] = ".clicktale.net"; // PRIVUS Tracker - 2009-09-28 BadURL_Parts[i++] = "ads\.js"; // PRIVUS Tracker - 2009-08-30 BadURL_WordStarts[i++] = "dcs\.gif"; // PRIVUS Tracker - 2009-09-19 To: BadDomains[i++] = ".clicktale.net"; // Tracker - 2009-10-26 BadURL_Parts[i++] = "ads\.js"; // Tracker - 2009-10-26 BadURL_WordStarts[i++] = "dcs\.gif"; // Tracker - 2009-10-26 Reason: These need to be there for Chrome, IE, Opera, and Safari users. 24. Action: Removed two of the number host rules from pornproxy. Date: 2009-Oct-28 04:26 UTC Removed: BadHostParts[i++] = "877"; BadHostParts[i++] = "900"; Reason: Rodney claimed they caused false positives. I don't have any but I don't have true negatives either. 25. Action: Seeing if browsers still allow hexadecimal IP addresses Added: BadHostWordStarts[i++] = "0x"; // PRIVUS Malware - 2009-10-27 Reason: I don't think browsers allow these any more. { 2010-01-23: Experimental rule; yielded nothing. } 26. Action: slight downgrading of URL start rule. From: BadURL_WordStarts[i++] = "eros"; To: BadURL_WordStarts[i++] = "eros[^e]"; // Malware - 2009-10-27 Reason: No significant loss in power - but no false positives in my PHTTPD flags either. 27. Action: Reduced the scope of the tracker rule From: BadHostParts[i++] = "tracker"; To: BadHostWordEnds[i++] = "tracker"; // Tracker - 2009-10-27 Reason: hacker-tracker.com & quotetracker.com It doesn't really help these two hosts but I can already tell where this is heading. But since it does whack out these hosts we will white-list them. 28. Action: Added some exclusions to the "tracker" rule which even though I downgraded the rule, these hosts still won't make it past the rule. Added: GoodDomains[i++] = "hacker-tracker.com"; // tracker - 2009-10-27 GoodDomains[i++] = "quotetracker.com"; // tracker - 2009-10-27 Reason: If I get any more I will just remove the "tracker" rule. We already have two other exclusions, securitytracker.com and versiontracker.com (which I just added a date to). When I say remove, it will just be a rule for ME. 29. Action: White list rule for false positive. Added: GoodDomains[i++] = "stock-anal.com"; // anal - 2009-10-28 Reason: "anal" rule over-ride. 30. Action: White list rule for security site. Added: GoodDomains[i++] = "kaspersky-labs.com"; // SECURITY - 2009-10-28 Reason: Security web site. 31. Action: Added an optional designation to a rule that caused Rodney some problems From: BadURL_Parts[i++] = "exploited"; To: BadURL_Parts[i++] = "exploited"; // YOUR CHOICE - 2009-10-28 Reason: Let social scientist types remove it with very little to no thought of it having too much of a negative impact. 32. Action: mozdev.org white-listed Added: GoodDomains[i++] = ".mozdev.org"; // block - 2009-10-30 Reason: block even though it is optional. 33. Action: Removed the BadDomains free.fr Date: 2009-Oct-31 14:07 UTC Removed: BadDomains[i++] = ".free.fr"; // YOUR CHOICE Reason: Most of the porn type hosts in the domain are handled quite easily by the other porn rules now. Also, if you enable the free rule you block it anyway. 34. Action: Broadened scope of utm rule Added: BadURL_Parts[i++] = "utm\.gif"; // Tracker - 2009-11-02 Reason: It is really in the URL that we need this. The problem is that they use more than the utm.gif file, and I have NO idea what the false positives are going to be like. But for example, EasyPrivacy has wgat we have for the host rule and __utm.gif URL start for the URL rule. If I need to, I will drop this rule to that same status. The __utm.js seems to be needed according to the EasyPrivacy person. But let's work with this one first. If it works out okay, THEN we can experiment with the JavaScript file. 35. Action: More Omniture specific script names Added: BadURL_WordStarts[i++] = "s_code_remote\.js"; // PRIVUS Tracker - 2009-11-02 BadURL_WordStarts[i++] = "omniture_code\.js"; // PRIVUS Tracker - 2009-11-02 Reason: If you strip these you strip the mechanism to even CALL the Omniture's *.2o7.net or sphere hosts. When these work well, it seems to peel all Omniture code out but more to the point, SOMETIMES IT SEEMS TO HANG THE HOST! IOW, These may be forever PRIVUS and may be very likely removed in the future. The s-code.js rule is in 21. { 2010-01-23: They are now public. } 36. Action: Added an PRIVUS rule - SHOULD EVERYBODY HAVE THIS? From: GoodDomains[i++] = "dashboard.godaddy.com"; // PRIVUS - 2009-09-30 To: GoodDomains[i++] = "dashboard.godaddy.com"; // 2009-11-02 Reason: It showed up in my phttp logs, but I will be DARNED if I can understand what is causing it! Originally I said: "Until I understand it, I cannot add it for everybody. There is NO rule I can point to that causes the problem. WHAT IS IT!?". I rescind it. I don't care what it stops. This rule is added and if I still have problems then godaddy.com itself will get white list status. The same goes for any other domain name controller mechanism. SO POINT THEM OUT IF YOU CAN FIND THEM! { 2010-01-23: Issue resolved by hackers trying to gain user accounts at GoDaddy via pattern squatting: GoodDomains[i++] = ".godaddy.com"; BadHostParts[i++] = "godaddy\.com"; These work because of what rules come when. If you are at ANYSUB.godaddy.com the first rule allows you access because it comes BEFORE the second one. But if you are given godaddy.com.elsewhere.co.uk or NewGoDaddy.com it doesn't pass muster with the first rule and then the second rule kicks in and stops the pretender. } 37. Action: Another WebTrends way of doing things. Added: BadURL_WordStarts[i++] = "webtrends_tag\.js"; // Tracker - 2009-11-03 Reason: Encountered it at foxreality.com which was blocked for months! Nobody complained. Now they can't do WebTrends blocking any more though. 38. Action: Added BadDomain rules to the pornproxy files ONLY Date: 2009-Nov-03 11:41 UTC Added: BadDomains[i++] = ".deluxepass.com"; // PORN - 2009-11-03 BadDomains[i++] = ".partie-privee.com"; // PORN - 2009-11-03 BadDomains[i++] = ".thumblogger.com"; // PORN - 2009-11-03 Reason: They had no malware, do no tracking and thus can only be put into the pornproxy file. 39. Action: Added BadDomain rules to ALL files Date: 2009-Nov-03 11:41 UTC Added: BadDomains[i++] = ".advance.net"; // AdServer - 2009-11-03 BadDomains[i++] = ".dynamic.dol.ru"; // AdServer - 2009-11-03 BadDomains[i++] = ".links.channelintelligence.com"; // Tracker - 2009-11-03 BadDomains[i++] = ".origin.channelintelligence.com"; // Tracker - 2009-11-03 BadDomains[i++] = ".rdr.channelintelligence.com"; // Tracker - 2009-11-03 BadDomains[i++] = ".pochta.ru"; // MalWare - 2009-11-03 Reason: Per Rodney's request but I have been thinking about the next change for some time as the number of malware hosts in the *.ru domain has dropped. 40. Action: Made the Russian rule optional Date: 2009-Nov-03 12:40 UTC From: BadDomains[i++] = ".ru"; // YOUR CHOICE - MalWare To: // BadDomains[i++] = ".ru"; // YOUR CHOICE - MalWare Reason: I have observed the count going down at both Airelle's hosts.rsk file and MalwareDomainList's hosts file. Another part of this is that now I am adding badly behaved Russian domains (see 39 previous). 02 Novembre 2009 UNresolved False Positives (HHH) ------------------------------------------------- NONE 02 Novembre 2009 RESOLVED False Positives (HHH) ----------------------------------------------- SEE THE ACTIONS IN THE ITEMS ABOVE FOR THE RESOLUTION OF ALL OUTSTANDING FALSE POSITIVES THAT WERE HANDLED. There were others, but they were for the optional opt-in rules.