I’ll write the code logically according to the control flow. I’ll start with the cli implementation, then move over to implementing propertysite and the crawler. After that html parser and data handler. I’ll make a minimal version of each first, so that they work and are tested, as a proof of concept, before I try to do anything fancy. Its an iterative process.

Here is a list of the core functionalities each module should have to function.

  1. CLI: accept the URL for a propertysite and pass it on to crawler and propertysite
    1. no options for differrent behaviours, in the beginning it should just be a port
  2. propertysite: the generalized abc and an implementation for remax
    1. listing class to house individual listing objects
    2. actual information on how the sites are put together, html structure and variable names
      1. retrieve by hand
  3. crawler: retrieves the first listing site as html
  4. htmlparser: is able to take a list of elements and output the contents of those elements
  5. datahandler: only geocoding, spits the errored rows out, prints the geocoded results

modules

cli

tests

Functionalities that need to be tested: both input and output,

propertysite

Is implemented first as an abstract base class which is then implemented in

Getting the html structure

I need to find out what the html structure of a listing page and the listinglist pages are, so that I can later parse them with HTML parser. I need to do this by hand the first time.

Listing page

My listing class look like this (this abc is then defined with site specific values):

    class Listing(abc.ABC):
        """ Information about specific listings """
        url = ""            # baseurl for all listings
        list_header = ""    # list header name
        header_html = ()    # html list header element structure
        attr_html = ()      # attribute list html element structure
        attr_keys = []      # attribute titles from html
        def __init__(self, id, attr_dict):
            self.id = id        # url + id -> listing location on website
            self.attr_dict = attr_dict # listing attributes with title and value

Of which we need the latter three class variablesfrom the html file ( I guess we would get the id attribute from there as well).

I downloaded a listing as html via firefox. The attributes list header is found at:

<div class="remax-ad-item-header" id="osoite_heading">

<h5>Perustiedot</h5>

</div>

# after which all of the list items are like this:

<div id="osoite">

<div class="remax-ad-item-body">

<div class="row-list-oe" id="osoite_heading-body">

<div class="row mx-n1">

<div class="col-12 col-md-5 list-label">Osoite:</div>

<div class="col-12 col-md-7 list-value">Markulantie 119 20320 Turku</div>

</div>

<div class="row mx-n1">

<div class="col-12 col-md-5 list-label">Kaupunginosa/kylä:</div>

<div class="col-12 col-md-7 list-value">Länsikeskus</div>

</div>

As I look at the HTML, it is obvious that the h5 header is the starting point from where I start parsing, then the list label is the main stop point, and the list value is the second stop point. Then the labels are searched for and the values are checked. With my comprehension I need both list label and value html as a tuple, or only the beginning part of the html tag, and just move onto the next line if </ is encountered. The second one is a bit less readable, I think, but nicer overall. So I will do that.

This is what my implemented class variables look like now:

        list_header = "<h5>Perustiedot</h5>"
        label_value_tag = ('<div class="col-12 col-md-5 list-label">',
                           '<div class="col-12 col-md-7 list-value">')

So I renamed the attr_html variable to label_value_tag, which is more descriptive.

Next up is populating the list of list labels I want to parse for. I have them as a text file.

I decided on having them as a constant in a constants.py file, since the list is way too big to house within code. I also got rid of the remax class in the propertysite file. Its now in its own file.

Christ, I foolishly opened vscode from another folder than i am used to and the relative imports exploded and I kept trying to debug it in a panic for almost an hour. Christ, thank goodness I figured it out, but that was all for me for today.

Listing list page html

I’ll do the same method as I did with the listing page with the listing list page. Get it from the web, inspect and add necessary features to the class.
I need from there:

  • links to listings
  • button element

The listing boxes are generated through a function:

    function outputListing(listing) {
        return `
		<div class="col-12 col-sm-6 col-lg-4">
			<a class="apartment-list-items" href="/kohde/${listing.url}">
				<div class="item-image">
					<img src="${listing.image}" loading="lazy" alt="${listing.short_location}, ${listing.street_address}">

...

So the element is <a class="apartment-list-items" href="/kohde/${listing.url}"> and we’ll use as the tag <a class="apartment-list-items" href="/kohde/ and parse for the listing url, we don’t need anything else from it, just repeating it until we hit the next button or the end. (The IDs could be stored in a set, because it doesn’t allow duplicates.)

There are 3 instances of the next button “lisää-painike” in the HTML, one for rental listings, one for property listings and one for business listings. We are only interested in the propertylistings. The other two are hidden if the rental or property listings are not shown. At this point I’m not sure how the crawler would handle this. Could I just tell the section name (eg property-button), and parse that section for the button, or do I have to jump through some hoops.

I’ll paste the full section here and get back to it later:


    <section id="apartment-listings" class="listings-section hidden init">
        <div class="container container-xl" id="list-controls">
            <div class="row align-items-sm-center pt-3 mt-3">
                <div class="col-12 col-sm-auto">
                    <h4 class="mb-sm-0" id="listing-count">
                        <span class="amount"></span> kohdetta                    </h4>
                </div>
                <div class="col-12 col-sm-auto ml-sm-auto">
                    <div class="d-flex align-items-center">
                        <div class="dropdown-select w-100">
                            <div class="dropdown-wrapper search-ordering">
                                <a class="selected-value" data-value="">Järjestä tulokset</a>
                                <div class="dropdown-container">
                                    <ul class="dropdown-items">
                                        <li data-value="uusimmat-ensin">Uusin ensin</li>
                                        <li data-value="vanhimmat-ensin">Vanhin ensin</li>
                                        <li data-value="halvimmat-ensin">Halvin ensin</li>
                                        <li data-value="kalleimmat-ensin">Kallein ensin</li>
                                        <li data-value="pienimmat-ensin">Pienin ensin</li>
                                        <li data-value="suurimmat-ensin">Suurin ensin</li>
                                        <li data-value="esittelyyn-tulevat-ensin">Esittelyyn tulevat ensin</li>
                                    </ul>
                                </div>
                            </div>
                        </div>

                        <!--div class="d-none d-md-inline-block ml-4 pl-3">
                            <div class="d-flex align-items-center">
                                <div class="checkbox-switch-label pr-3">KARTTA</div><label class="checkbox-switch"
                                    id="property-checkbox-switch" style="background-color: rgb(255, 255, 255);"><input
                                        name="switch" type="checkbox"><span class="checkbox-toggle"
                                        id="property-checkbox-toggle"
                                        style="background-color: rgb(0, 61, 165);"></span></label>
                            </div>
                        </div-->
                    </div>
                </div>
            </div>
        </div>
        <section id="property-listings">
            <div class="container container-xl" id="properties-list">
                <div class="row row-xl-35"></div>
            </div>
            <!--div id="hakuvahti-hissi"><a href="/hakuvahti/" target="_blank"><h3>Tilaa hakuvahti</h3></a></div-->
        </section>
        <div class="loadmore hidden">
            <div class="aligncenter text-center stk-block">
				<a class="stk-link stk-button linkki-painike readmore"><span class="stk-button__inner-text">Katso lisää</span></a>
            </div>
		</div>
    </section>

I found the call that gets made when the next button is pressed from the network tab in devtools! Its gets the content as json per page according to the search filters. I can just use this with wget to get brief information about the sites (all I need is the ID, which is included). One problem is that this particular call doesn’t respond with all of the listings, but only listings per page. That isn’t that big of a deal, since I can just loop throug the queries with &page=i until the response value is empty!

https://remax.fi/wp-content/themes/blocksy-child/property_search_LINEAR.php?property-type=asunnot&realty-type=&bedrooms=&showings-from=&showings-to=&location=turku&price_min=&price_max=&living_area_m2_min=&living_area_m2_max=&lot_area_min=&lot_area_max=&buildyear_min=&buildyear_max=&location=turku&page=8

This eases my workload significantly. And I will try to access the call for the listing page aswell!

  • I think the property page is just a static site which is sent directly from the server to the browser, so the listing information isn’t queried indidually.
  • This isn’t such a problem, but the more you know!

Next idea is about how the interface to the application should be like. Should I pass the php query or the first url to the system. If I jut pass the first url, how do I programatically access the php query?

  • The original url doesn’t contain the search filters, like I thought they did (oikotie has them). Which means the crawler would have to set the search parameters dynamically, and frankly, aint nobody got time for that.
  • Ergo, for Remax I’ll just use the php query in the cli api. Its not pretty but its the quickest way.

As the php query returns json, I’ll need to parse that.

And a few hours later it is done as well as other stuff! I got so far to be able to populate the listing_list with listing type objects that have correct ids. I’m not quite sure what else to do right now (except write thorough tests), other than to move to crawler implementation.

crawler

Implementation for the crawler should be even more streamlined as I found the php search url.

For listinglist:

  • We need to make a function that sends the php search out with incrementing pagenumbers
  • function that saves all of that data to one file (appends each page)
    • I have the functionality for reading the data already For listing site:
  • a function that forms the proper url that can be then requested
  • a function that requests the site contents
  • a function that cycles through all of the requests
  • a function that saves a request content.

After these steps we should be able to move onto htmlparser

The response content length is 2,44 kB when there is no content from the php search. Its about 5 kB when there is. Mayb I could use the size of the resulting file to exit the while loop cycling through listinglist pages.

  • A better way is to print the output of wget to stdout and save it with tee to the correct file. This way we can read the file contents at the source, without external checking logic. wget -q -O - "link.com" | tee newfile.html

    datahandler

I already touched on this in the other sections as datahandler IO class affects the json and html methodology.

htmlparser

main

tests

I used this article for a tutorial on the unit testing framework medium.com, I used this one too: pytest-with-eric.com

I know I wanted to use unittest, but I think I will use pytest instead, since it is advertised with better scalability to larger applications.

Points:

  • test single units, a function or method, in isolation
    • test functionality based on expected inputs and edge cases
  • Use Arrange, Act, Assert pattern
  • mock external dependencies
  • descriptive test names
  • setup and teardown for reliable outcomes
    • using @pytest.fixture wrapper
  • I’ll mirror the source directory structure in my test files

After reading these instructions I moved to the guide in pytest’s documentation: documentation

  • need to figure out fixtures next
  • https://docs.pytest.org/en/latest/reference/fixtures.html#fixtures I’ll start today with reading on the other features of pytest before moving onto testing and coding otherwise
  • fixtures can aid the arrange phaze of a test by inititalizing reused variables, reducing boilerplate

the parametrizicing fixture might prove useful:

@pytest.mark.parametrize("input_value, expected_output", [(1, 2), (5, 6), (10, 11)])
def test_increment(input_value: int, expected_output: int) -> None:
    assert input_value + 1 == expected_output

from: https://docs.pytest.org/en/latest/explanation/types.html

  • important: pytest caches fixtures (one instance per fixture per test suite), so changing values of variables inside fixtures that are used in multiple tests can lead to unexpected results. https://docs.pytest.org/en/latest/how-to/fixtures.html#fixture-scopes
<
Previous Post
levylaatikko.02 rakennusprosessi ja pohdinta
>
Next Post
mikrotaso.01