Replace Alexa with Similarweb (#6634)

pull/6651/head
Michael Loßin 2 years ago committed by GitHub
parent f94bf30a09
commit d597d4b524
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -28,19 +28,10 @@ jobs:
name: Check Facebook handles
command: ruby ./tests/facebook.rb
- run:
name: Check Alexa ranking
command: bundle exec ruby ./tests/alexa.rb
- run:
name: Check Similarweb ranking
command: bundle exec ruby ./tests/similarweb.rb
- save_cache:
key: awis-cache
paths:
- "/tmp/alexa/*"
- save_cache:
key: similarweb-cache
paths:

@ -53,7 +53,7 @@ body:
Leave any unmet requirements unchecked, and add any additional information or questions in the "Additional information" section below.
options:
- label: I have checked that the site meets 2fa.directory's [contributing guidelines and site criteria](https://github.com/2factorauth/twofactorauth/blob/master/CONTRIBUTING.md)
- label: The site is ranked within the top 200,000 sites on [Alexa](https://www.alexa.com/siteinfo/) or has a substantial social media following.
- label: The site is ranked within the top 200,000 sites on [Similarweb](https://www.similarweb.com/) or has a substantial social media following.
- label: The site does not contain or promote content that violates 2fa.directory's [excluded categories and websites guidelines](https://github.com/2factorauth/twofactorauth/blob/master/EXCLUSION.md), including pornographic, discriminatory, or unlawful content.
- label: The issue I'm creating is not a duplicate of an existing [issue](https://github.com/2factorauth/twofactorauth/issues).
required: true

@ -53,7 +53,7 @@ body:
Leave any unmet requirements unchecked, and add any additional information or questions in the "Additional information" section below.
options:
- label: I have checked that the site meets 2fa.directory's [contributing guidelines and site criteria](https://github.com/2factorauth/twofactorauth/blob/master/CONTRIBUTING.md)
- label: The site is ranked within the top 200,000 sites on [Alexa](https://www.alexa.com/siteinfo/) or has a substantial social media following.
- label: The site is ranked within the top 200,000 sites on [Similarweb](https://www.similarweb.com/) or has a substantial social media following.
- label: The site does not contain or promote content that violates 2fa.directory's [excluded categories and websites guidelines](https://github.com/2factorauth/twofactorauth/blob/master/EXCLUSION.md), including pornographic, discriminatory, or unlawful content.
- label: The issue I'm creating is not a duplicate of an existing [issue](https://github.com/2factorauth/twofactorauth/issues).
required: true
@ -63,4 +63,4 @@ body:
- type: textarea
attributes:
label: Additional information
description: If you have any additional information to provide, please do so below.
description: If you have any additional information to provide, please do so below.

@ -34,7 +34,7 @@ body:
- It no longer meets 2fa.directory's contributing guidelines and site criteria.
- It contains content that violates 2fa.directory's excluded categories and websites guidelines.
- It has been acquired or absorbed by another site, and the new site is already listed on 2fa.directory.
- It no longer has verifiable 2FA documentation and is no longer ranked within the top 200,000 sites on Alexa.
- It no longer has verifiable 2FA documentation and is no longer ranked within the top 200,000 sites on Similarweb.
- Other (please describe below).
validations:
required: true

@ -18,9 +18,8 @@ jobs:
with:
path: |
/tmp/iso-*.txt
/tmp/alexa
/tmp/similarweb
key: tmp-files-${{ hashFiles('/tmp/alexa/*','/tmp/similarweb/*')}}
key: tmp-files-${{ hashFiles('/tmp/similarweb/*')}}
- name: Change nameserver
run: echo "nameserver 1.1.1.1" > /etc/resolv.conf
- name: Validate JSON structure

@ -23,8 +23,8 @@ are stored in folders corresponding to each of those entries in their own
the file to be under 2.5 kB.
3. **HTTPS links**: All sites that support HTTPS should also be linked with an
HTTPS address.
4. **Alexa top 200K**: A new site that is not already listed has to be within the
Alexa top 200,000 ranking. You can check the ranking of a site [here][alexa].
4. **Similarweb top 200K**: A new site that is not already listed has to be within the
Similarweb top 200,000 ranking. You can check the ranking of a site [here][similarweb].
5. **No 2FA providers**: We do not list 2FA providers, such as [Authy][authy], [Duo][duo] or
[Google Authenticator][googleauthenticator].
6. **Be Awesome**: You need to be awesome. That is all.
@ -360,7 +360,7 @@ Many sites are now offering passwordless authentication, which replace the passw
[img]: img/
[svgo]: https://github.com/svg/svgo
[tinypng]: https://tinypng.com/
[alexa]: https://www.alexa.com/siteinfo
[similarweb]: https://www.similarweb.com/
[authy]: https://authy.com/
[duo]: https://duo.com/
[googleauthenticator]: https://github.com/google/google-authenticator

@ -1,51 +0,0 @@
#!/usr/bin/env ruby
# frozen_string_literal: true
require 'uri'
require 'net/http'
require 'json'
require 'fileutils'
# rubocop:disable Metrics/AbcSize
def fetch_from_api(site)
url = URI("https://awis.api.alexa.com/api?Action=UrlInfo&ResponseGroup=Rank&Output=json&Url=#{site}")
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
headers = { 'x-api-key' => ENV['ALEXA_API_KEY'], 'Accept' => 'application/json' }
response = http.request(Net::HTTP::Get.new(url, headers))
raise("(#{response.code}) Request failed.") unless response.code == '200'
rank = JSON.parse(response.body)['Awis']['Results']['Result']['Alexa']['TrafficData']['Rank']
store_cache(site, rank)
raise("#{site} doesn't have an Alexa ranking") if rank.nil?
raise("#{site} is ranked above the maximum rank of 200K") if rank.to_i > 200_000
end
# rubocop:enable Metrics/AbcSize
def fetch_from_cache(site)
path = "/tmp/alexa/#{site}"
return false unless File.exist?(path)
rank = File.read(path)
raise("#{site} doesn't have an Alexa ranking") if rank.eql?('')
raise("#{site} is ranked above the maximum rank of 200K") if rank.to_i > 200_000
rank
end
def store_cache(site, rank)
FileUtils.mkdir_p('/tmp/alexa/')
File.open("/tmp/alexa/#{site}", 'w') { |file| file.write rank }
end
status = 0
# Fetch changes
diff = `git diff origin/master...HEAD entries/ | sed -n 's/^+.*"domain"[^"]*"\\(.*\\)".*/\\1/p'`
# Strip and loop through diff
diff.split("\n").each do |site|
fetch_from_api(site) unless fetch_from_cache(site)
rescue StandardError => e
puts "\e[31m#{e.message}\e[39m"
status = 1
end
exit(status)

@ -57,4 +57,4 @@ rescue StandardError => e
puts "\e[31m#{e.message}\e[39m"
status = 1
end
exit(status) if Time.now.to_i > 1_651_363_200
exit(status)

Loading…
Cancel
Save