Skip to content

Commit 066e224

Browse files
ptkatoptkato
committed
Initial commit
0 parents  commit 066e224

13 files changed

+285
-0
lines changed

‎.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
.vscode/
2+
dist-newstyle/

‎LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2021 Patrick Augusto
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
This is a daft retail store scrapper, I created it because I'm too lazy to check up on the availability of video cards, which with all the pricing craze, keep going out of stock bloody quick.

‎exe/FileParser.hs

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
module FileParser
2+
( parseSources
3+
) where
4+
5+
parseSources :: [String] -> IO [[String]]
6+
parseSources retailers = do
7+
sources <- traverse (readFile . ("retailers/" ++)) retailers
8+
return $ map lines sources

‎exe/Main.hs

+41
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
module Main where
2+
3+
import FileParser
4+
5+
import Requisitioner
6+
import Scrapper
7+
import Types
8+
9+
import Control.Concurrent
10+
import Control.Monad
11+
import Data.Foldable
12+
import System.Random
13+
14+
import qualified Data.ByteString.Lazy as LBS
15+
import qualified Data.Text.Lazy as LT
16+
import qualified Data.Text.Lazy.Encoding as LTE
17+
import qualified Network.Wreq.Session as WS
18+
19+
main :: IO ()
20+
main = do
21+
sources <- parseSources ["kabum", "pichau", "terabyteshop"]
22+
putStrLn "\n" >> mainGo sources
23+
24+
mainGo
25+
:: [[String]]
26+
-> IO ()
27+
mainGo fs = do
28+
forever $ do
29+
sess <- WS.newSession
30+
31+
for_ fs $ \retailers ->
32+
forkIO . for_ retailers $ \source -> do
33+
let target = targetfy source
34+
res <- LTE.decodeLatin1 <$> request sess target
35+
traverse_ putStrLn $ scrapper target res
36+
37+
pause <- randomRIO (10000000, 20000000)
38+
threadDelay pause
39+
40+
threadDelay 900000000
41+
return ()

‎retailers/kabum

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
https://www.kabum.com.br/produto/109716/placa-de-v-deo-gigabyte-amd-radeon-rx-5600-xt-gaming-oc-6gb-gddr6-gv-r56xtgaming-oc-6gd
2+
https://www.kabum.com.br/produto/109984/placa-de-v-deo-asrock-amd-radeon-rx-5600-xt-phantom-gaming-d3-oc-6gb-ddr6-90-ga1vzz-00uanf
3+
https://www.kabum.com.br/produto/109986/placa-de-v-deo-asrock-amd-radeon-rx-5600-xt-challenger-d-oc-6gb-ddr6-90-ga1xzz-00uanf
4+
https://www.kabum.com.br/produto/110385/placa-de-v-deo-xfx-amd-radeon-rx-5600-xt-thicc-iii-ultra-6gb-gddr6-rx-56xt6tbd8-
5+
https://www.kabum.com.br/produto/113296/placa-de-v-deo-gigabyte-amd-radeon-rx-5600-xt-windforce-oc-6g-6gb-gddr6-rev-1-0-gv-r56xtwf2oc-6gd
6+
https://www.kabum.com.br/produto/115727/placa-de-v-deo-powercolor-amd-radeon-rx-5600-xt-6gb-gddr6-axrx-5600xt-6gbd6-3dhv2-oc
7+
https://www.kabum.com.br/produto/116162/placa-de-v-deo-gigabyte-radeon-windforce-rx-5600-xt-amd-ati-6gb-gv-r56xtwf2-6gd-rev-1-0-
8+
https://www.kabum.com.br/produto/128200/placa-de-v-deo-gigabyte-amd-radeon-rx-5600-xt-windforce-6g-gddr6-rev-2-0-gv-r56xtwf2-6gd
9+
https://www.kabum.com.br/produto/133031/-placa-de-v-deo-power-color-amd-rx-5600-xt-6gb-gddr6-axrx-5600xt-6gbd6-3dhe-oc
10+
https://www.kabum.com.br/produto/133032/placa-de-v-deo-power-color-amd-rx-5600-xt-6gb-gddr6-axrx-5600xt-6gbd6-3dhr-oc

‎retailers/pichau

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-xfx-radeon-rx-5600-xt-6gb-tchicc-ii-pro-192-bit-rx-56xt6dfd6?utm_source=escorregaopreco
2+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-gigabyte-radeon-rx-5600-xt-6gb-windforce-oc-192-bit-gv-r56xtwf2oc-6gd?utm_source=escorregaopreco
3+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-xfx-radeon-rx-5600-xt-6gb-thicc-iii-ultra-192-bit-rx-56xt6tbd8?utm_source=escorregaopreco
4+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-gigabyte-radeon-rx-5600-xt-6gb-gaming-oc-192-bit-gv-r56xtgaming-oc-6gd?utm_source=escorregaopreco
5+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-sapphire-radeon-rx-5600-xt-6gb-pulse-256-bit-11296-01-20g?utm_source=escorregaopreco
6+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-asus-radeon-rx-5600-xt-6gb-gddr6-rog-strix-oc-192-bit-rog-strix-rx5600xt-t6g-gaming?utm_source=escorregaopreco
7+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-asrock-radeon-rx-5600-xt-6gb-phantom-gaming-d2-oc-192-bit-90-ga1wzz-00uanf?utm_source=escorregaopreco
8+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-asrock-radeon-rx-5600-xt-6gb-challenger-oc-edition-192-bit-90-ga1xzz-00uanf?utm_source=escorregaopreco
9+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-msi-radeon-rx-5600-xt-6gb-mech-oc-128-bit-912-v381-227?utm_source=escorregaopreco
10+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-asus-radeon-rx-5600-xt-6gb-gddr6-oc-192-bit-dual-rx5600xt-t6g-evo?utm_source=escorregaopreco
11+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-sapphire-radeon-rx-5600-xt-be-6gb-pulse-192-bit-11296-05-20g?utm_source=escorregaopreco
12+
https://www.pichau.com.br/hardware/placa-de-video/placa-de-video-asrock-radeon-rx-5600-xt-phantom-gaming-d3-oc-6gb-192-bit-90-ga1vzz-00uanf?utm_source=escorregaopreco

‎retailers/terabyteshop

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
https://www.terabyteshop.com.br/produto/15570/placa-de-video-msi-radeon-rx-5600-xt-gaming-x-dual-6gb-gddr6-192bit-912-v381-245
2+
https://www.terabyteshop.com.br/produto/15571/placa-de-video-msi-radeon-rx-5600-xt-gaming-mx-dual-6gb-gddr6-192bit-912-v381-256
3+
https://www.terabyteshop.com.br/produto/14352/placa-de-video-sapphire-pulse-amd-radeon-rx-5600-xt-be-6gb-gddr6
4+
https://www.terabyteshop.com.br/produto/13995/placa-de-video-powercolor-radeon-navi-rx-5600-xt-dual-oc-6gb-gddr6-axrx-5600xt-6gbd6-3dhv2oc
5+
https://www.terabyteshop.com.br/produto/13270/placa-de-video-asrock-amd-radeon-rx-5600-xt-challenger-d-oc-6gb-gddr6-90-ga1xzz-00uanf
6+
https://www.terabyteshop.com.br/produto/13271/placa-de-video-asrock-amd-radeon-rx-5600-xt-phantom-gaming-d2-oc-6gb-gddr6-90-ga1wzz-00uanf
7+
https://www.terabyteshop.com.br/produto/13272/placa-de-video-asrock-amd-radeon-rx-5600-xt-phantom-gaming-d3-oc-6gb-gddr6-90-ga1vzz-00uanf

‎shell.nix

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{ pkgs ? import <nixpkgs> { } }:
2+
with pkgs;
3+
4+
let
5+
haskellDeps = ps: with ps; [
6+
zlib
7+
];
8+
haskellEnv = haskellPackages.ghcWithPackages haskellDeps;
9+
in mkShell {
10+
buildInputs = [
11+
haskellEnv
12+
];
13+
}

‎src/Requisitioner.hs

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
{-# LANGUAGE OverloadedStrings #-}
2+
3+
module Requisitioner
4+
( request
5+
) where
6+
7+
import Types
8+
9+
import Control.Lens
10+
import Network.Wreq (defaults, header, responseBody, responseHeaders)
11+
import Network.Wreq.Session
12+
import Network.Wreq.Types
13+
14+
import qualified Data.ByteString.Char8 as BS8
15+
import qualified Data.ByteString.Lazy as LBS
16+
17+
headers' :: Options
18+
headers' = defaults
19+
& header "User-Agent" .~ ["Mozilla/5.0 (X11; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"]
20+
& header "Accept" .~ ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"]
21+
& header "Accept-Language" .~ ["en-IE,en-AU;q=0.8,en-GB;q=0.7,en;q=0.5,fr-FR;q=0.3,pt-BR;q=0.2"]
22+
& header "Accept-Encoding" .~ ["gzip, deflate, br"]
23+
& header "Connection" .~ ["keep-alive"]
24+
& header "Referer" .~ ["https://www.google.com/"]
25+
& header "Upgrade-Insecure-Requests" .~ ["1"]
26+
& header "Pragma" .~ ["no-cache"]
27+
& header "Cache-Control" .~ ["no-cache"]
28+
& header "TE" .~ ["Trailers"]
29+
30+
request :: Session -> Target -> IO LBS.ByteString
31+
request sess t = do
32+
res <- getWith
33+
(headers' & header "Host" .~ [BS8.pack . show $ _host t])
34+
sess
35+
(show t)
36+
37+
return $ res ^. responseBody

‎src/Scrapper.hs

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
{-# LANGUAGE OverloadedStrings #-}
2+
3+
module Scrapper
4+
( scrapper
5+
) where
6+
7+
import Types
8+
9+
import Control.Lens ((^?), (^..), ix, only)
10+
import Text.Taggy
11+
import Text.Taggy.Lens
12+
13+
import qualified Data.Text.Lazy as LT
14+
15+
scrapper :: Target -> LT.Text -> Maybe String
16+
scrapper t@(Target Kabum _) = kabum (show t)
17+
scrapper t@(Target Pichau _) = pichau (show t)
18+
scrapper t@(Target Terabyteshop _) = terabyteshop (show t)
19+
20+
kabum :: String -> LT.Text -> Maybe String
21+
kabum link page =
22+
link <$ page ^? html . elements
23+
. allAttributed (ix "alt" . only "produto_disponivel")
24+
25+
pichau :: String -> LT.Text -> Maybe String
26+
pichau link page =
27+
link <$ page ^? html . elements
28+
. allAttributed (ix "class" . only "stock unavailable")
29+
30+
terabyteshop :: String -> LT.Text -> Maybe String
31+
terabyteshop link page =
32+
link <$ page ^? html . elements
33+
. allAttributed (ix "title" . only "Produto dispon\195\173vel no estoque.")

‎src/Types.hs

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
{-# LANGUAGE RecordWildCards #-}
2+
module Types
3+
( Target(..)
4+
, Host(..)
5+
, targetfy
6+
) where
7+
8+
import Data.String
9+
import Text.Parsec
10+
11+
data Host = Kabum | Pichau | Terabyteshop deriving Eq
12+
13+
data Target = Target
14+
{ _host :: Host
15+
, _resource :: String
16+
}
17+
18+
instance Show Host where
19+
show Kabum = "https://www.kabum.com.br/"
20+
show Pichau = "https://www.pichau.com.br/"
21+
show Terabyteshop = "https://www.terabyteshop.com.br/"
22+
23+
instance Show Target where
24+
show (Target h r) = show h ++ r
25+
26+
instance IsString Host where
27+
fromString "https://www.kabum.com.br/" = Kabum
28+
fromString "https://www.pichau.com.br/" = Pichau
29+
fromString "https://www.terabyteshop.com.br/" = Terabyteshop
30+
fromString s = error s
31+
32+
instance IsString Target where
33+
fromString = targetfy
34+
35+
targetfy :: String -> Target
36+
targetfy s = let appendSlash x = x ++ "/" in
37+
case runParser (sepBy (many $ noneOf "/") (string "/")) () "" s of
38+
Right xs -> do
39+
let _host = fromString . concatMap appendSlash . take 3 $ xs
40+
_resource = init . concatMap appendSlash . drop 3 $ xs
41+
42+
Target {..}
43+
Left x -> error $ show x
44+
45+
46+
47+
48+

‎video-scrapper.cabal

+52
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
cabal-version: 3.0
2+
name: video-scrapper
3+
version: 0.1.0.0
4+
build-type: Simple
5+
license: MIT
6+
7+
extra-source-files:
8+
README.md
9+
shell.nix
10+
retailers/*
11+
12+
library
13+
hs-source-dirs: src
14+
15+
exposed-modules:
16+
Requisitioner,
17+
Scrapper,
18+
Types
19+
20+
other-extensions:
21+
OverloadedStrings
22+
RecordWildCards
23+
24+
build-depends:
25+
base,
26+
bytestring,
27+
text,
28+
parsec,
29+
wreq,
30+
lens,
31+
taggy,
32+
taggy-lens
33+
34+
default-language: Haskell2010
35+
36+
executable video-scrapper-exe
37+
main-is: Main.hs
38+
39+
other-modules:
40+
FileParser
41+
42+
build-depends:
43+
base,
44+
bytestring,
45+
text,
46+
wreq,
47+
random,
48+
video-scrapper
49+
50+
hs-source-dirs: exe
51+
52+
default-language: Haskell2010

0 commit comments

Comments
 (0)