One Statistics Professor Was Just Banned By Google: Here Is His Story

http://investingchannel.com/article...r-Was-Just-Banned-By-Google-Here-Is-His-Story

Merits aside, this is another reminder that it's dangerous to depend too much on Google.

 

Looks like Google is on a slippery slope

 

The companies control the narrative.

 

With all due respect this raises a lot of questions, like how is it this guy did not see, on the balance of probabilities, what happens when corps turn into monsters such as google - and how is it he didn't operate with more care to have backups of his data so he could at least bounce back and relocate elsewhere?

Anyone who puts this amount of trust in the internet is eventually in for a rude awakening.

 

Its not just Google its all across the board at every level. If they all continue unopposed we are going to be looking at a black box internet where the user has no control over anything except their choice of phony competitor providers that jointly control it all.
All the signs are already there.

 

You mean like now.

 

Right. It's hard to imagine how this guy trusted Google so blindly. But I've seen similar stories every month or two, where various providers have nuked accounts with no recourse. And there are similar sob stories about building a new business that relies on some other business, such as an API. If the API goes away for whatever reason, or access gets restricted, you're toast. Ditto with accounts on social media

 

Yes or a corporation that doesn't like what that application does buys the company and phases it out.
IMO a lot of mistakes have been made and most of them were caused by those who claimed they were making the internet better than it was before with their new technologies while supporting a hidden agenda that had nothing to do with making the internet better and everything to do with making it profitable for themselves.
The upside is, it doesn't have to remain that way. Port 80 is only 1of 65565 available ports, meaning anyone who wants to could start a project to make a new net with a constitution setting out pre defined principles instead of allowing the kind of open evolution that created the corporate cesspool the internet is today.

 

Yes, an open Internet is an amazing vision. And the prospect of authoritarian control is chilling. Even if the authorities are driven by profit maximization, rather than ideology.

That's why we need systems designed such that censorship and coercion are impossible. Not just those with power don't censor because they're so liberal and all. No, we need systems that are just plain uncensorable by design. And given the Internet that we have, that means overlay networks.

 

This takes a bit of a leap. So here goes. The profit is had. Men's minds as toy things is the pleasure of the wicked.

 

Nice

 

Yes I agree with that entirely

 

Agreed.

And that's why I hold on to the internet lightly so to speak. What is paradoxical is that the best brains out there still can't see the simple fact of how fickle the internet is - or more specifically the monsters behind it, but who wants to hear there's more to this than mere $'s.

 

Download your webmail on a desktop email client. That way, if your online account goes dead, you have all your data in your hands. You just make a new webmail account, and copy your data into it.

 

Yes, and getting rid of DNS (or at least lessening dependency of it). No seriously. The only reason we have the thing in the first place is because human mind is lousy remembering IP addresses.
So many things would go away (DNS hijacking/spoofing, DNS-level censoring, DNS leaks .....).

I have one crazy idea but will see within few weeks if it's worth the trouble...

 

Are you gonna do the IP address pinning thing we talked about?
Here's my input regarding security featutes and I dont know how much of it is practical but first time you visit a site pin the cert fingerprint and the IP address to the domain name and save to file.
Subsequent visits to that domain checks your pinned list before doing a dns request. If that domain is in your pinned list then connect using the IP. This might prevent the SSL error you would expect if you typed the IP into the address bar instead of the URL.
That would probably take some work to integrate it into favorites etc but would be a great security feature.
Then the next thing is to block third party connections and make a button that allows the user to view the blocked connections and enable any that turn out to be necessary for the site to work.
That should block tracking analyctics etc.
Then lastly is to ensure the browser can only use good tls ciphers and ideally show the user which cipher is in use.

 

Yes, I will investigate the IP pinning for the cyberdragon but I have to give it priority 3. Have to fix the bugs in imagesecret and try my little crazy project for getting rid of completely (or almost) of DNS first.

Actually, now that i think of it. Before I put last version of cyberdragon to hold, I did make some experimental, private version of it that had some nice feature, like experimental automatic iframe sandboxing (even if the web developer did not use sandbox HTML attribute for his/her pages) and improved tracking list speed with updated, 300 000+ URL/domain tracker list.

Maybe time to dig it up again soon and dust it .....

 

The current DNS system is absurd given how cheap memory & bandwidth is now. You could easily do blocks of top 100k+ sites, and then distribute updates or do a real DNS request if that failed. The lists and update distribution all being cryptographically signed, and giving away zero information, plus having much better MiTM protection. Top it off with associated cert verification, and that's pretty good.

Which function "should" be in the browser - but we know how they behave, they're part of the problem (not yours Stefan!).

 

Yes the DNS system is one of the security loopholes that the corporate devs would rather ignore. That together with CA and TLS issues.
All of which could be mitigated with a few minor changes to the way browsers handle them.

 

It's like you had read my mind

As we speak my poor rented VPS server is mapping the whole Internet. Yes the whole d*mn internet.

Doing parallel mapping of whole countries and saving all the hostname/IP combos that have been registered in that country into XZ compressed CSV files at first, and later maybe using relational database (maybe sqlite)

I started carefully on 11th of August and only mapped few countries at the time so that not to exceed my bandwith limit.
Latest of parallel mapping processes is now in MX (mexico) so about lil over half of the countries are almost all completed (really big ones like china and india still mapping).

Largest fully completed file is Argentina:

AR.cvs.xz
18 MB xz compressed (630 MB uncompressed)
13 975 593 hostname/IP combinations

I estimate that at this rate that Im doing and starting mapping processes carefully, the whole big I is mapped in maybe two or three weeks more and the XZ compressed space requirements in total maybe somewhere around 3 GB (best case) to 24 GB (worst case). And this with my poor VPS server with 2 Gb uplink and 8 cores (think what renting Amazon EC2 could do , they offer free trials btw)

So yea, you could carry whole internet in your pocket
And then write a special "DNS" proxy program that is only running on your phone/tablet/desktop/laptop etc.... at address 127.0.0.1 and listening any outgoing 53 port request, catch them, do quickly checking from XZ compressesed CVS file (or maybe better from sqlite database file) the requested hostname and spit the IP address back to requesting software.
Without doing any real DNS request at all.

Also, you could completely block all outgoing DNS request (just block port 53) to all except that "DNS" proxy program that would only do real DNS request in the last possible case as fallback, for example if the lists needs updating but for some reason can't contact update server(s).
Another advantage is that any rogue software running in your system and trying to do their own DNS would not get anywhere. They would need to play nice and talk to that gatekeeper "DNS" proxy software.

So hardware is no problem at all. Plenty of bandwith, memory, CPU power and hard disk space.

Only problem is, how often the lists should be updated by contacting update server(s) or if, distributed model, p2p nodes ?
Of course only lists that have changed would be needed to download (simple sha1 checksum compare of the remote files and local)

EDIT: And of course, nothing prevents to break those mapped files into smaller, more manageable blocks. So instead of whole countries, map just by top-level domain (.com, .net .au etc....)

 

Stefan this will probably go down as the most insane thing I have ever read on this forum. Your own portable DNS server !!!
That is freakin genius.

 

Visionary might have been kinder!

Of course, the source of the information and sound cryptography need to be handled with care, on the other hand, we trust the Linux repos, and it would be relatively easy to cross correlate data from different sources to expose any manipulation of the data at any point.

It would be interesting to know the churn rate of addresses - I suspect it's relatively low for the more popular sites, which is what most of the point is (as well as taking back control).

 

I think he knows what I meant by insane, in that slang context it means an idea so far out of the box that even the word brilliant is not enough.

 

No worries RockLobster I got it

But lets do some more calculations to see if this crazy idea is practical.

As surely everyone here knows, the IPv4 addresses that we humans present in XXX.XXX.XXX.XXX form
are really just nothing more than 32-bit unsigned integers to computers.

So the maximum possible number of public IP addresses is 2^32 or 4 294 967 296
Whoah! Lot's of addresses!

But wait, it get's better:

According to this, if we substract the reserved IP addresses (that is, address ranges that are used mostly in LANs) out from that big block we only have 3 706 452 992 addresses left.

https://stackoverflow.com/questions/2437169/what-is-the-total-amount-of-public-ipv4-addresses

Still quite alot.
But it get's even better!

According to this, there are only 326 million registered domain names as of 2016.
https://investor.verisign.com/releasedetail.cfm?releaseid=980215

And it makes perfect sense, not every IP address out there have a domain name registered to it.

So, storing 326 million hostname/IP address combos....
How much storage it would need if saving the whole thing as uncompressed CSV-file(s) ?

If each line in the CSV-file(s) would have the following format

"hostname_here(max 253 bytes length)" + "," (1 byte) + "XXX.XXX.XXX.XXX" (15 bytes) + "LF" (1 byte) (2 bytes "CRLF" in Windows)

Then 270 (271 for windows) bytes per line.
That times 326 million ... 88020000000 (88346000000 windows) bytes.

That's just lousy 81 (82 windows) GB !!! Uncompressed!
And that's on a assumption that everyone of those 326 domainnames would be the max. length (253) long which clearly is not the case.

Gosh, I could take my old 128 GB USB-stick and stuff it with all Internet hostname/IP address info.

How about if instead of CSV plaintext we use binary format?

Then it could be something like this:

1 byte length + hostname(253 bytes again) + dword (32-bit unsigned integer 4 bytes)

258 bytes.
That times 326 ... about 79 GB
So not much saving, so not much point using it, unless .....
What if we change the format to use sha1 checksum to make a blindingly fast search functionality ?(comparing long strings is notoriously time consuming with computers)

Like this:

sha1 hash presentation of the hostname (20 bytes) + dword (32-bit unsigned integer 4 bytes)

24 bytes !!!
And size ... about 8 GB ...
Oh my god!

The whole d*mn internet, in 8 GB file and fast search functionality (convert given hostname to SHA1 hash and search from list)

It's doable folks, it's doable

Only thing is, the mapping of this takes annoyingly long.

The way it now works, I have extracted the IP address pools, in CIDR format, from the RIR records for every country (they take only about 3.5 MB in all).
And saving each to their own file with country code name (like DE.cidr) and then feeding those files to parallel nmap processes and piping the output to xz compressor.

What Im not sure of, even tought the nmapping is done in parallel, I don't know if the actual mapping process uses asynchronous DNS or does it just goes like: check hostname1, wait, check hostname2, wait .....

I have started writing my own little program that will get the DNS PTR records (mapping of IP address to hostname) by using asynchronous DNS requests and then going to compare the performance against nmap.

EDIT: Ah, there was a slight mistake in calculations.
Sure, all the domain names could be stored into 8 GB binary file but I did not take into account sub-domains. Well, even if all the 3 706 452 992 possible public IP addresses had all (sub-)domain name assigned to them it would still take only about 83 GB to store it all
(( 3 706 452 992 * 24)/1073741824) = 82,845.