✨ Ajoute le parser Safti

Fix #20

README.md

@@ -1,85 +1,46 @@
-<p align="center"><img src="https://res.cloudinary.com/dtfbvvkyp/image/upload/v1566331377/laravel-logolockup-cmyk-red.svg" width="400"></p>
+# My Home collection ![language](https://img.shields.io/badge/language-laravel-blue.svg) ![issues](https://img.shields.io/github/issues-raw/Chouchen/Shikiryu_backup) ![ci](https://ci.canhelpme.com/build-status/image/7?branch=main&label=PHPCensor&style=flat-square)
 
-<p align="center">
+> Because I need to keep some house in mind
-<a href="https://travis-ci.org/laravel/framework"><img src="https://travis-ci.org/laravel/framework.svg" alt="Build Status"></a>
-<a href="https://packagist.org/packages/laravel/framework"><img src="https://poser.pugx.org/laravel/framework/d/total.svg" alt="Total Downloads"></a>
-<a href="https://packagist.org/packages/laravel/framework"><img src="https://poser.pugx.org/laravel/framework/v/stable.svg" alt="Latest Stable Version"></a>
-<a href="https://packagist.org/packages/laravel/framework"><img src="https://poser.pugx.org/laravel/framework/license.svg" alt="License"></a>
-</p>
 
-## About Laravel
+App to save house sale ads
 
-Laravel is a web application framework with expressive, elegant syntax. We believe development must be an enjoyable and creative experience to be truly fulfilling. Laravel takes the pain out of development by easing common tasks used in many web projects, such as:
+## :books: Table of Contents
 
-- [Simple, fast routing engine](https://laravel.com/docs/routing).
+- [Installation](#package-installation)
-- [Powerful dependency injection container](https://laravel.com/docs/container).
+- [Usage](#rocket-usage)
-- Multiple back-ends for [session](https://laravel.com/docs/session) and [cache](https://laravel.com/docs/cache) storage.
+- [Support](#hammer_and_wrench-support)
-- Expressive, intuitive [database ORM](https://laravel.com/docs/eloquent).
+- [Contributing](#memo-contributing)
-- Database agnostic [schema migrations](https://laravel.com/docs/migrations).
+- [License](#scroll-license)
-- [Robust background job processing](https://laravel.com/docs/queues).
-- [Real-time event broadcasting](https://laravel.com/docs/broadcasting).
 
-Laravel is accessible, powerful, and provides tools required for large, robust applications.
+## :package: Installation
 
-## Learning Laravel
+### Requirements
 
-Laravel has the most extensive and thorough [documentation](https://laravel.com/docs) and video tutorial library of all modern web application frameworks, making it a breeze to get started with the framework.
+ * git
+ * composer
+ * yarn or npm
+ * a database (SQLite, MySQL, …)
 
-If you don't feel like reading, [Laracasts](https://laracasts.com) can help. Laracasts contains over 1500 video tutorials on a range of topics including Laravel, modern PHP, unit testing, and JavaScript. Boost your skills by digging into our comprehensive video library.
+### Then install this script
 
-## Laravel Sponsors
+```sh
+git clone https://git.shikiryu.com/Shikiryu/MyHomeCollection.git
+composer install
+yarn install
+```
 
-We would like to extend our thanks to the following sponsors for funding Laravel development. If you are interested in becoming a sponsor, please visit the Laravel [Patreon page](https://patreon.com/taylorotwell).
+## :rocket: Usage
 
-### Premium Partners
+Create an account, add an ad, use the app !
 
-- **[Vehikl](https://vehikl.com/)**
+## :hammer_and_wrench: Support
-- **[Tighten Co.](https://tighten.co)**
-- **[Kirschbaum Development Group](https://kirschbaumdevelopment.com)**
-- **[64 Robots](https://64robots.com)**
-- **[Cubet Techno Labs](https://cubettech.com)**
-- **[Cyber-Duck](https://cyber-duck.co.uk)**
-- **[Many](https://www.many.co.uk)**
-- **[Webdock, Fast VPS Hosting](https://www.webdock.io/en)**
-- **[DevSquad](https://devsquad.com)**
 
-### Community Sponsors
+Please [open an issue](https://git.shikiryu.com/Shikiryu/MyHomeCollection/issues/new) for support.
 
-<a href="https://op.gg"><img src="http://opgg-static.akamaized.net/icon/t.rectangle.png" width="150"></a>
+## :memo: Contributing
 
-- [UserInsights](https://userinsights.com)
+Please contribute using [Github Flow](https://guides.github.com/introduction/flow/). Create a branch, add commits, and [open a pull request](https://github.com/Chouchen/Shikiryu_Backupleonard-henriquez/readme-boilerplateleonard-henriquez/readme-boilerplate/compare/).
-- [Fragrantica](https://www.fragrantica.com)
-- [SOFTonSOFA](https://softonsofa.com/)
-- [User10](https://user10.com)
-- [Soumettre.fr](https://soumettre.fr/)
-- [CodeBrisk](https://codebrisk.com)
-- [1Forge](https://1forge.com)
-- [TECPRESSO](https://tecpresso.co.jp/)
-- [Runtime Converter](http://runtimeconverter.com/)
-- [WebL'Agence](https://weblagence.com/)
-- [Invoice Ninja](https://www.invoiceninja.com)
-- [iMi digital](https://www.imi-digital.de/)
-- [Earthlink](https://www.earthlink.ro/)
-- [Steadfast Collective](https://steadfastcollective.com/)
-- [We Are The Robots Inc.](https://watr.mx/)
-- [Understand.io](https://www.understand.io/)
-- [Abdel Elrafa](https://abdelelrafa.com)
-- [Hyper Host](https://hyper.host)
-- [Appoly](https://www.appoly.co.uk)
-- [云软科技](http://www.yunruan.ltd/)
 
-## Contributing
+## :scroll: License
 
-Thank you for considering contributing to the Laravel framework! The contribution guide can be found in the [Laravel documentation](https://laravel.com/docs/contributions).
+[Creative Commons Attribution NonCommercial (CC-BY-NC)](https://tldrlegal.com/license/creative-commons-attribution-noncommercial-(cc-nc)) © [Chouchen](https://github.com/Chouchen/)
-
-## Code of Conduct
-
-In order to ensure that the Laravel community is welcoming to all, please review and abide by the [Code of Conduct](https://laravel.com/docs/contributions#code-of-conduct).
-
-## Security Vulnerabilities
-
-If you discover a security vulnerability within Laravel, please send an e-mail to Taylor Otwell via [taylor@laravel.com](mailto:taylor@laravel.com). All security vulnerabilities will be promptly addressed.
-
-## License
-
-The Laravel framework is open-sourced software licensed under the [MIT license](https://opensource.org/licenses/MIT).

app/Console/Commands/ParseLinkCommand.php

@@ -3,7 +3,9 @@
 namespace App\Console\Commands;
 
 use App\Parser;
+use GuzzleHttp\Exception\InvalidArgumentException;
 use Illuminate\Console\Command;
+use function GuzzleHttp\json_encode;
 
 class ParseLinkCommand extends Command
 {
@@ -39,7 +41,11 @@ class ParseLinkCommand extends Command
     public function handle()
     {
         $parser = Parser::factory($this->argument('url'));
-        $parser->parse();
+        try {
+            $this->info(json_encode($parser->parse(), true));
+        } catch (InvalidArgumentException $e) {
+            $this->error($e->getMessage());
+        }
 
         return 0;
     }

app/Exceptions/UnknownParser.php

@@ -0,0 +1,8 @@
+<?php
+
+namespace App\Exceptions;
+
+class UnknownParser extends Exception
+{
+
+}

app/Parser.php

@@ -2,6 +2,7 @@
 
 namespace App;
 
+use App\Exceptions\UnknownParser;
 use GuzzleHttp\Client;
 use Illuminate\Support\Facades\Config;
 
@@ -36,7 +37,11 @@ abstract class Parser
             }
         }
 
-        return null;
+        throw new UnknownParser(sprintf(
+            'Can\'t find an url in «%s» file with the following URL «%s». Please update the file accordingly.',
+            'parser.php',
+            $url
+        ));
     }
 
     /**

app/Parser/Orpi.php

@@ -0,0 +1,192 @@
+<?php
+
+namespace App\Parser;
+
+use App\ParsedHome;
+use App\Parser;
+use GuzzleHttp\Exception\InvalidArgumentException;
+use Symfony\Component\DomCrawler\Crawler;
+use function GuzzleHttp\json_decode;
+
+/**
+ * Class Orpi
+ * @package App\Parser
+ */
+class Orpi extends Parser
+{
+    /**
+     * @inheritDoc
+     */
+    public function parse(): ParsedHome
+    {
+        $request = $this->client->get($this->url);
+        $body = $request->getBody()->getContents();
+        $crawler = new Crawler($body);
+        $parsedHome = new ParsedHome();
+        /**
+         * Orpi ads can be parsed 2 ways :
+         *  * sometimes, a JSON is included in the page so it's just a reading/feeding object
+         *  * else, we must crawl the webpage…
+         */
+        $data_estate = $crawler->filter('[data-estate]');
+        if ($data_estate->count() > 0) {
+            return $this->parseJSON($parsedHome, $crawler);
+        }
+
+        return $this->parseHTML($parsedHome, $crawler);
+    }
+
+    /**
+     * @param int $score
+     *
+     * @return string
+     */
+    private function calculateDPE($score)
+    {
+        if (empty($score)) {
+            return 'Inconnu';
+        }
+        if ($score <= 50) {
+            return 'A';
+        }
+        if ($score >= 51 && $score <= 90) {
+            return 'B';
+        }
+        if ($score >= 91 && $score <= 150) {
+            return 'C';
+        }
+        if ($score >= 151 && $score <= 230) {
+            return 'D';
+        }
+        if ($score >= 231 && $score <= 330) {
+            return 'E';
+        }
+        if ($score >= 331 && $score <= 450) {
+            return 'F';
+        }
+        if ($score > 450) {
+            return 'G';
+        }
+
+        return 'Inconnu';
+    }
+
+    /**
+     * @param $score
+     *
+     * @return string
+     */
+    private function calculateGES($score)
+    {
+        if (empty($score)) {
+            return 'Inconnu';
+        }
+        if ($score <= 5) {
+            return 'A';
+        }
+        if ($score >= 6 && $score <= 10) {
+            return 'B';
+        }
+        if ($score >= 11 && $score <= 20) {
+            return 'C';
+        }
+        if ($score >= 21 && $score <= 35) {
+            return 'D';
+        }
+        if ($score >= 36 && $score <= 55) {
+            return 'E';
+        }
+        if ($score >= 56 && $score <= 80) {
+            return 'F';
+        }
+        if ($score > 80) {
+            return 'G';
+        }
+
+        return 'Inconnu';
+    }
+
+    /**
+     * @param \App\ParsedHome                       $parsed_home
+     * @param \Symfony\Component\DomCrawler\Crawler $crawler
+     *
+     * @return \App\ParsedHome
+     */
+    private function parseJSON(ParsedHome $parsed_home, Crawler $crawler)
+    {
+        $data_estate = $crawler->filter('[data-estate]');
+        try {
+            $json_data = json_decode($data_estate->attr('data-estate'), true);
+            $parsed_home->price = $json_data['price'];
+            $parsed_home->city = $json_data['city']['name'];
+            $parsed_home->surface = $json_data['surface'];
+            $parsed_home->garden_surface = $json_data['lotSurface'];
+            $parsed_home->rooms = $json_data['nbRooms'];
+            $parsed_home->description = $json_data['longAd'];
+            $parsed_home->title = $json_data['seo']['metaTitle'];
+            $parsed_home->map = ['lat' => $json_data['latitude'], 'lng' => $json_data['longitude']];
+            $parsed_home->pictures = $json_data['imagesFull'];
+            $parsed_home->energy = $this->calculateDPE($json_data['consumptionValue']);
+            $parsed_home->ges = $this->calculateGES($json_data['emissionValue']);
+
+            return $parsed_home;
+        } catch (InvalidArgumentException $e) {
+            return $this->parseHTML($parsed_home, $crawler);
+        }
+    }
+
+    /**
+     * @param \App\ParsedHome                       $parsed_home
+     * @param \Symfony\Component\DomCrawler\Crawler $crawler
+     *
+     * @return \App\ParsedHome
+     */
+    private function parseHTML(ParsedHome $parsed_home, Crawler $crawler)
+    {
+        $ad = $crawler->filter('article');
+        $first_section = $ad->children()->first();
+        $second_section = $ad->children()->eq(1);
+        $third_section = $ad->children()->eq(2);
+        $parsed_home->description = $second_section->filter('.o-container')->children()->eq(1)->text();
+        $second_section->filter('.c-badge__text')->each(static function (Crawler $detail, $i) use (&$parsed_home) {
+            $detail_text = $detail->text();
+            if (mb_strpos($detail_text, 'Terrain') === 0) {
+                $parsed_home->garden_surface = mb_substr($detail_text, 8, -2);
+            }
+
+            if (mb_strpos($detail_text, 'pièces') !== false) {
+                $parsed_home->rooms = (int)$detail_text;
+            }
+        });
+        $h1 = $first_section->filter('h1');
+        $parsed_home->title = $h1->children()->first()->text();
+        $parsed_home->surface = (int)$h1->children()->eq(2)->text();
+        $parsed_home->city = $h1->children()->eq(4)->text();
+        $parsed_home->price = (int)str_replace(' ', '', $first_section->filter('.u-h1')->text());
+        $third_section->filter('.c-dpe')->each(static function (Crawler $detail, $i) use (&$parsed_home) {
+            $abbr = $detail->filter('abbr');
+            if ($abbr->count() > 0) {
+                if ($detail->attr('c-dpe--ges') !== null) {
+                    $parsed_home->ges = $abbr->text();
+                } elseif ($detail) {
+                    $parsed_home->energy = $abbr->text();
+                }
+            }
+        });
+
+        $request = $this->client->get($this->url.'/photos/');
+        $body = $request->getBody()->getContents();
+        $crawler = new Crawler($body);
+        $parsed_home->pictures = $crawler
+            ->filter('.u-cover')
+            ->each(static function (Crawler $node, $i) {
+                if (strtolower($node->nodeName()) === 'img') {
+                    return $node->attr('src');
+                }
+
+                return null;
+            });
+
+        return $parsed_home;
+    }
+}

app/Parser/OuestFrance.php

@@ -24,7 +24,8 @@ class OuestFrance extends Parser
         $parsedHome->surface = (int)str_replace(' ', '', $details->eq(2)->filter('strong')->text());
         $parsedHome->garden_surface = (int)str_replace(' ', '', $details->eq(3)->filter('strong')->text());
         $parsedHome->rooms = (int)str_replace(' ', '', $details->eq(4)->filter('strong')->text());
-        $parsedHome->energy = $crawler->filter('#dpeCateg > strong')->text();
+        $dpeCateg = $crawler->filter('#dpeCateg > strong');
+        $parsedHome->energy = $dpeCateg->count() === 1 ? $dpeCateg->text() : '';
 //        $parsedHome->city = ?
 //        $parsedHome->map = ?
         $parsedHome->pictures = $crawler

app/Parser/Safti.php

@@ -0,0 +1,67 @@
+<?php
+
+namespace App\Parser;
+
+use App\ParsedHome;
+use App\Parser;
+use NumberFormatter;
+use Symfony\Component\DomCrawler\Crawler;
+
+class Safti extends Parser
+{
+
+    /**
+     * @inheritDoc
+     */
+    public function parse(): ParsedHome
+    {
+        $request = $this->client->get($this->url);
+        $body = $request->getBody()->getContents();
+        $crawler = new Crawler($body);
+        $parsed_home = new ParsedHome();
+
+        $property_single = $crawler->filter('[data-testid="real-estate"]');
+
+        $number_formatter = new NumberFormatter('en', NumberFormatter::DECIMAL);
+        $currency_formatter = new NumberFormatter('en', NumberFormatter::CURRENCY);
+        $currency = 'EUR';
+
+        $parsed_home->title = $property_single->filter('h1')->text();
+        $parsed_home->price = $currency_formatter->parseCurrency($property_single->filter('.property__price')->text(), $currency);
+        $parsed_home->city = $property_single->children()->children('div')->eq(1)->filter('p.h4')->text();
+        $parsed_home->description = $crawler->filter('[data-testid="real-estate-annonce-single-description"]')->text();
+
+        $property__additionals = $crawler->filter('.property__additionals');
+        $energies = $property__additionals->filter('.energetic-indicator');
+        if ($energies->count() > 0) {
+            $parsed_home->energy = substr($energies->eq(0)->text(), 0, 1);
+            $parsed_home->ges = substr($energies->eq(1)->text(), 0, 1);
+        }
+
+        $crawler
+            ->filter('.property__informations .mobile-extends > div .property__informations__element')
+            ->each(static function (Crawler $property____information, $i) use (&$parsed_home, $number_formatter) {
+                $name = trim($property____information->filter('i')->text());
+                $value = trim($property____information->filter('b')->text());
+                switch ($name) {
+                    case 'Pièces :':
+                        $parsed_home->rooms = (int)$value;
+                        break;
+                    case 'Surface habitable :':
+                        $parsed_home->surface = $number_formatter->parse($value);
+                        break;
+                    case 'Terrain :':
+                        $parsed_home->garden_surface = $number_formatter->parse($value);
+                        break;
+                    default:
+//                        break;
+                }
+            });
+
+        $parsed_home->pictures = $crawler->filter('[data-testid="real-estate-mosaic-photo"]')->filter('img')->each(static function($img) {
+            return $img->attr('src');
+        });
+
+        return $parsed_home;
+    }
+}

app/Providers/AppServiceProvider.php

@@ -2,7 +2,6 @@
 
 namespace App\Providers;
 
-use Illuminate\Pagination\Paginator;
 use Illuminate\Support\ServiceProvider;
 use Illuminate\Support\Str;
 
@@ -35,6 +34,5 @@ class AppServiceProvider extends ServiceProvider
             }
             return sprintf('%s m²', number_format($surface, 0, ',', ' '));
         });
-        Paginator::useBootstrap();
     }
 }

composer.json

@@ -8,27 +8,27 @@
     ],
     "license": "MIT",
     "require": {
-        "php": "^7.3.0",
+        "php": "^7.2.5",
         "ext-json": "*",
         "absmoca/leboncoin": "dev-master",
-        "artesaos/seotools": "^0",
+        "artesaos/seotools": "^0.18.0",
         "emanueleminotto/simple-html-dom": "^1.5",
-        "fabpot/goutte": "^4.0",
+        "fabpot/goutte": "^3.1",
         "fideloper/proxy": "^4.2",
         "fruitcake/laravel-cors": "^2.0",
-        "guzzlehttp/guzzle": "^7.0.1",
+        "guzzlehttp/guzzle": "^6.3",
-        "laravel/framework": "^8.0",
+        "laravel/framework": "^7.0",
         "laravel/tinker": "^2.0",
-        "laravel/ui": "^3.0",
+        "laravel/ui": "^2.1",
         "spatie/laravel-feed": "^2.7",
         "spatie/laravel-query-builder": "^2.8"
     },
     "require-dev": {
-        "facade/ignition": "^2.3.6",
+        "facade/ignition": "^2.0",
         "fzaninotto/faker": "^1.9.1",
         "mockery/mockery": "^1.3.1",
-        "nunomaduro/collision": "^5.0",
+        "nunomaduro/collision": "^4.1",
-        "phpunit/phpunit": "^9.0"
+        "phpunit/phpunit": "^8.5"
     },
     "config": {
         "optimize-autoloader": true,
@@ -42,9 +42,7 @@
     },
     "autoload": {
         "psr-4": {
-            "App\\": "app/",
+            "App\\": "app/"
-            "Database\\Factories\\": "database/factories/",
-            "Database\\Seeders\\": "database/seeders/"
         },
         "classmap": [
             "database/seeds",

config/parser.php

@@ -3,8 +3,10 @@
 use App\Parser\ImmobilierNotaires;
 use App\Parser\LannionImmo;
 use App\Parser\LBC;
+use App\Parser\Orpi;
 use App\Parser\OuestFrance;
 use App\Parser\Pap;
+use App\Parser\Safti;
 use App\Parser\SeLoger;
 
 return [
@@ -14,6 +16,8 @@ return [
         'pap.fr'                => Pap::class,
         'ouestfrance-immo.com'  => OuestFrance::class,
         'lannion.immo'          => LannionImmo::class,
-        'immobilier.notaires.fr'=> ImmobilierNotaires::class
+        'immobilier.notaires.fr'=> ImmobilierNotaires::class,
+        'orpi.com'              => Orpi::class,
+        'safti.fr'              => Safti::class,
     ],
 ];

database/factories/UserFactory.php

@@ -1,7 +1,5 @@
 <?php
 
-namespace Database\Factories;
-
 /** @var \Illuminate\Database\Eloquent\Factory $factory */
 
 use App\User;

database/migrations/2020_09_25_090124_update_failed_jobs_laravel8.php

@@ -1,32 +0,0 @@
-<?php
-
-use Illuminate\Database\Migrations\Migration;
-use Illuminate\Database\Schema\Blueprint;
-use Illuminate\Support\Facades\Schema;
-
-class UpdateFailedJobsLaravel8 extends Migration
-{
-    /**
-     * Run the migrations.
-     *
-     * @return void
-     */
-    public function up()
-    {
-        Schema::table('failed_jobs', function (Blueprint $table) {
-            $table->string('uuid')->after('id')->nullable()->unique();
-        });
-    }
-
-    /**
-     * Reverse the migrations.
-     *
-     * @return void
-     */
-    public function down()
-    {
-        Schema::table('failed_jobs', static function (Blueprint $table) {
-            $table->dropColumn('uuid');
-        });
-    }
-}

database/seeds/DatabaseSeeder.php

@@ -1,7 +1,5 @@
 <?php
 
-namespace Database\Seeders;
-
 use Illuminate\Database\Seeder;
 
 class DatabaseSeeder extends Seeder

public/index.php

@@ -9,10 +9,6 @@
 
 define('LARAVEL_START', microtime(true));
 
-if (file_exists(__DIR__.'/../storage/framework/maintenance.php')) {
-    require __DIR__.'/../storage/framework/maintenance.php';
-}
-
 /*
 |--------------------------------------------------------------------------
 | Register The Auto Loader