ooni.nettests manipulation Package

ooni.nettests.manipulation.captiveportal Module

class ooni.nettests.manipulation.captiveportal.CaptivePortal[source]

Bases: ooni.templates.httpt.HTTPTest, ooni.templates.dnst.DNSTest

Compares content and status codes of HTTP responses, and attempts to determine if content has been altered.

author = 'Isis Lovecruft'
check_0x20_to_auth_ns(*args, **kwargs)[source]

Resolve a 0x20 DNS request for hostname over hostname’s authoritative nameserver(s), and check to make sure that the capitalization in the 0x20 request matches that of the response. Also, check the serial numbers of the SOA (Start of Authority) records on the authoritative nameservers to make sure that they match.

If sample_size is given, a random sample equal to that number of authoritative nameservers will be queried; default is 5.

compare_random_hostnames(*args, **kwargs)[source]

Get hostname_count number of random hostnames with SLD length of hostname_length, and then attempt DNS resolution. If no arguments are given, default to three hostnames of 32 bytes each. These random hostnames should resolve to NXDOMAIN, except in the case where a user is presented with a captive portal and remains unauthenticated, in which case the captive portal may return the address of the authentication page.

If the cardinality of the intersection of the set of resolved random hostnames and the single element control set ([‘NXDOMAIN’]) are equal to one, then DNS properly resolved.

Returns true if only NXDOMAINs were returned, otherwise returns False with the relative complement of the control set in the response set.

control(*args, **kwargs)[source]

Compares the content and status code of the HTTP response for experiment_url with the control_result and control_code respectively. If the status codes match, but the experimental content and control_result do not match, fuzzy matching is enabled to determine if the control_result is at least included somewhere in the experimental content. Returns True if matches are found, and False if otherwise.

description = 'Captive Portal Test'
dns_resolve(*args, **kwargs)[source]

Resolves hostname(s) though nameserver to corresponding address(es). hostname may be either a single hostname string, or a list of strings. If nameserver is not given, use local DNS resolver, and if that fails try using 8.8.8.8.

dns_resolve_match(*args, **kwargs)[source]

Resolve experiment_hostname, and check to see that it returns an experiment_address which matches the control_address. If they match, returns True and experiment_address; otherwise returns False and experiment_address.

get_auth_nameservers(*args, **kwargs)[source]

Many CPs set a nameserver to be used. Let’s query that nameserver for the authoritative nameservers of hostname.

The equivalent of: $ dig +short NS ooni.nu

get_random_hostname(length=None)[source]

Returns a random hostname with SLD of specified length. If length is unspecified, length=32 is used.

These should all resolve to NXDOMAIN. If they actually resolve to a box that isn’t part of a captive portal that would be rather interesting.

get_random_url_safe_string(length)[source]

Returns a random url-safe string of specified length, where 0 < length <= 256. The returned string will always start with an alphabetic character.

google_dns_cp_test(*args, **kwargs)[source]

Google Chrome resolves three 10-byte random hostnames.

hostname_to_0x20(hostname)[source]

MaKEs yOur HOsTnaME lOoK LiKE THis.

For more information, see: D. Dagon, et. al. “Increased DNS Forgery Resistance Through 0x20-Bit Encoding”. Proc. CSS, 2008.

http_content_match_fuzzy_opt(*args, **kwargs)[source]

Makes an HTTP request on port 80 for experimental_url, then compares the response_content of experimental_url with the control_result. Optionally, if the fuzzy parameter is set to True, the response_content is compared with a regex of the control_result. If the response_content from the experimental_url and the control_result match, returns True with the HTTP status code and headers; False, status code, and headers if otherwise.

http_fetch(*args, **kwargs)[source]

Parses an HTTP url, fetches it, and returns a response object.

http_status_code_match(experiment_code, control_code)[source]

Compare two HTTP status codes, returns True if they match.

http_status_code_no_match(experiment_code, control_code)[source]

Compare two HTTP status codes, returns True if they do not match.

ms_dns_cp_test(*args, **kwargs)[source]

Microsoft “phones home” to a server which will always resolve to the same address.

name = 'captiveportal'
requiresRoot = False
requiresTor = False
run_vendor_dns_tests(*args, **kwargs)[source]

Run the vendor DNS tests.

run_vendor_tests(*args, **kwargs)[source]

These are several vendor tests used to detect the presence of a captive portal. Each test compares HTTP status code and content to the control results and has its own User-Agent string, in order to emulate the test as it would occur on the device it was intended for. Vendor tests are defined in the format: [exp_url, ctrl_result, ctrl_code, ua, test_name]

test_captive_portal(*args, **kwargs)[source]

Runs the CaptivePortal(Test).

CONFIG OPTIONS

If “do_captive_portal_vendor_tests” is set to “true”, then vendor specific captive portal HTTP-based tests will be run.

If “do_captive_portal_dns_tests” is set to “true”, then vendor specific captive portal DNS-based tests will be run.

If “check_dns_requests” is set to “true”, then Ooni-probe will attempt to check that your DNS requests are not being tampered with by a captive portal.

If “captive_portal” = “yourfilename.txt”, then user-specified tests will be run.

Any combination of the above tests can be run.

usageOptions

alias of UsageOptions

version = '0.3'
class ooni.nettests.manipulation.captiveportal.UsageOptions[source]

Bases: twisted.python.usage.Options

optParameters = [['asset', 'a', None, 'Asset file'], ['experiment-url', 'e', 'http://google.com/', 'Experiment URL'], ['user-agent', 'u', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6', 'User agent for HTTP requests']]

ooni.nettests.manipulation.daphne Module

class ooni.nettests.manipulation.daphne.Daphn3ClientFactory[source]

Bases: twisted.internet.protocol.ClientFactory

buildProtocol(addr)[source]
clientConnectionFailed(reason, connector)[source]
clientConnectionLost(reason, connector)[source]
mutation = [0, 0]
protocol

alias of Daphn3Protocol

startedConnecting(connector)[source]
steps = None
class ooni.nettests.manipulation.daphne.Daphn3ClientProtocol[source]

Bases: ooni.kit.daphn3.Daphn3Protocol

nextStep()[source]
class ooni.nettests.manipulation.daphne.daphn3Args[source]

Bases: twisted.python.usage.Options

optFlags = [['pcap', 'c', 'Specify that the input file is a pcap file'], ['yaml', 'y', 'Specify that the input file is a YAML file (default)']]
optParameters = [['host', 'h', '127.0.0.1', 'Target Hostname'], ['port', 'p', 57003, 'Target port number']]
class ooni.nettests.manipulation.daphne.daphn3Test[source]

Bases: ooni.nettest.NetTestCase

description = 'Bisects the censors fingerprint by mutating the given input packets.'
inputFile = ['file', 'f', None, 'Specify the pcap or YAML file to be used as input to the test']
inputProcessor(filename)[source]

step_idx is the step in the packet exchange ex. [.X.] are packets sent by a client or a server

client: [.1.] [.3.] [.4.] server: [.2.] [.5.]

mutation_idx: is the sub index of the packet as in the byte of the packet at the step_idx that is to be mutated

name = 'Daphn3'
requiresRoot = False
requiresTor = False
steps = None
test_daphn3()[source]
usageOptions

alias of daphn3Args

ooni.nettests.manipulation.dns_spoof Module

ooni.nettests.manipulation.http_header_field_manipulation Module

class ooni.nettests.manipulation.http_header_field_manipulation.HTTPHeaderFieldManipulation[source]

Bases: ooni.templates.httpt.HTTPTest

It performes HTTP requests with request headers that vary capitalization towards a backend. If the headers reported by the server differ from the ones we sent, then we have detected tampering.

author = 'Arturo Filast\xc3\xb2'
check_for_tampering(data)[source]

Here we do checks to verify if the request we made has been tampered with. We have 3 categories of tampering:

  • total when the response is not a json object and therefore we were not

able to reach the ooniprobe test backend

  • request_line_capitalization when the HTTP Request line (e.x. GET /

HTTP/1.1) does not match the capitalization we set.

  • header_field_number when the number of headers we sent does not match

with the ones the backend received

  • header_name_capitalization when the header field names do not match

those that we sent.

  • header_field_value when the header field value does not match with the

one we transmitted.

description = 'Checks if the HTTP request the server sees is the same as the one that the client has created.'
get_headers()[source]
get_random_caps_headers()[source]
name = 'HTTP Header Field Manipulation'
processResponseBody(data)[source]
randomizeUA = False
requiredOptions = ['backend']
requiredTestHelpers = {'backend': 'http-return-json-headers'}
requiresRoot = False
requiresTor = False
setUp()[source]
test_get_random_capitalization()[source]
usageOptions

alias of UsageOptions

version = '0.1.5'
class ooni.nettests.manipulation.http_header_field_manipulation.UsageOptions[source]

Bases: twisted.python.usage.Options

optParameters = [['backend', 'b', None, 'URL of the backend to use for sending the requests'], ['headers', 'h', None, 'Specify a yaml formatted file from which to read the request headers to send']]
ooni.nettests.manipulation.http_header_field_manipulation.random_capitalization(string)[source]

ooni.nettests.manipulation.http_host Module

class ooni.nettests.manipulation.http_host.HTTPHost[source]

Bases: ooni.templates.httpt.HTTPTest

This test performs various manipulations of the HTTP Host header field and attempts to detect which filter bypassing strategies will work against the censor.

Usually this test should be run with a list of sites that are known to be blocked inside of a particular network to assess which filter evasion strategies will work.

author = 'Arturo Filast\xc3\xb2'
check_for_censorship(body, test_name)[source]

XXX this is to be filled in with either a domclass based classified or with a rule that will allow to detect that the body of the result is that of a censored site.

description = 'Tests a variety of different filter bypassing techniques based on the HTTP Host header field.'
inputFile = ['file', 'f', None, 'List of hostnames to test for censorship']
inputProcessor(filename=None)[source]

This inputProcessor extracts domain names from urls

name = 'HTTP Host'
randomizeUA = False
requiredOptions = ['backend']
requiredTestHelpers = {'backend': 'http-return-json-headers'}
requiresRoot = False
requiresTor = False
setUp()[source]
test_filtering_add_tab_to_host(*args, **kwargs)[source]
test_filtering_of_subdomain(*args, **kwargs)[source]
test_filtering_prepend_newline_to_method(*args, **kwargs)[source]
test_filtering_via_fuzzy_matching(*args, **kwargs)[source]
test_send_host_header(*args, **kwargs)[source]

Stuffs the HTTP Host header field with the site to be tested for censorship and does an HTTP request of this kind to our backend.

We randomize the HTTP User Agent headers.

usageOptions

alias of UsageOptions

version = '0.2.4'
class ooni.nettests.manipulation.http_host.UsageOptions[source]

Bases: twisted.python.usage.Options

optParameters = [['backend', 'b', None, 'URL of the test backend to use. Should be listening on port 80 and be a HTTPReturnJSONHeadersHelper'], ['content', 'c', None, 'The file to read from containing the content of a block page']]

ooni.nettests.manipulation.http_invalid_request_line Module

class ooni.nettests.manipulation.http_invalid_request_line.HTTPInvalidRequestLine[source]

Bases: ooni.templates.tcpt.TCPTest

The goal of this test is to do some very basic and not very noisy fuzzing on the HTTP request line. We generate a series of requests that are not valid HTTP requests.

Unless elsewhere stated ‘Xx’*N refers to N*2 random upper or lowercase ascii letters or numbers (‘XxXx’ will be 4).

authors = 'Arturo Filast\xc3\xb2'
check_for_manipulation(response, payload)[source]
description = 'Performs out of spec HTTP requests in the attempt to trigger a proxy error message.'
name = 'HTTP Invalid Request Line'
requiredOptions = ['backend']
requiredTestHelpers = {'backend': 'tcp-echo'}
requiresRoot = False
requiresTor = False
setUp()[source]
test_random_big_request_method()[source]

This generates a request that looks like this:

Xx*512 / HTTP/1.1

test_random_invalid_field_count()[source]

This generates a request that looks like this:

XxXxX XxXxX XxXxX XxXxX

This may trigger some bugs in the HTTP parsers of transparent HTTP proxies.

test_random_invalid_method()[source]

We test sending data to a TCP echo server listening on port 80, if what we get back is not what we have sent then there is tampering going on. This is for example what squid will return when performing such request:

HTTP/1.0 400 Bad Request Server: squid/2.6.STABLE21 Date: Sat, 23 Jul 2011 02:22:44 GMT Content-Type: text/html Content-Length: 1178 Expires: Sat, 23 Jul 2011 02:22:44 GMT X-Squid-Error: ERR_INVALID_REQ 0 X-Cache: MISS from cache_server X-Cache-Lookup: NONE from cache_server:3128 Via: 1.0 cache_server:3128 (squid/2.6.STABLE21) Proxy-Connection: close
test_random_invalid_version_number()[source]

This generates a request that looks like this:

GET / HTTP/XxX

usageOptions

alias of UsageOptions

version = '0.2'
class ooni.nettests.manipulation.http_invalid_request_line.UsageOptions[source]

Bases: twisted.python.usage.Options

optParameters = [['backend', 'b', None, 'The OONI backend that runs a TCP echo server'], ['backendport', 'p', 80, 'Specify the port that the TCP echo server is running (should only be set for debugging)']]

ooni.nettests.manipulation.traceroute Module