@czolnowski opened this Pull Request on July 29th 2014 Contributor

Add offset and limit parameter to API methods which is used to fetching sites.

@czolnowski commented on July 29th 2014 Contributor
@mattab commented on August 1st 2014 Owner

Thanks for PR!

is there reason you added offset/paging to several APIs ie getSitesWithAdminAccess, getSitesWithViewAccess, getSitesWithAtLeastViewAccess ? it seems that the Manage Websites screen in #3838 only uses getSitesWithAdminAccess so maybe the parameter could be added only to this API?

@julienmoumne commented on August 6th 2014 Member

After discussing the SitesManager scalability issue during the meet-up, we concluded client side paging could be implemented as a first step.

If we later find client side paging does not scale enough, server side paging could be considered.

@czolnowski commented on August 6th 2014 Contributor

We tested SitesManager with 20.000 sites and backend doesn't respond, because of memory limit. For bigger instances of piwik we have implement backend limiting and offseting, as well filtering too.

@mattab commented on August 6th 2014 Owner

@czolnowski was there a memory error from server? because loading 20k websites in an array sounds cheap on memory eg. could fit in 200kb?

@czolnowski commented on August 25th 2014 Contributor

@mattab: Problem is when you want to print this using ResponseBuilder. It cause memory error.

@mattab commented on August 25th 2014 Owner

can you paste the memory error with backtrace? maybe we can make it less memory hungry as it shouldn't cause a memory error I think.

@czolnowski commented on August 25th 2014 Contributor

Here is one of logs from nginx:

2014/08/25 12:41:35 [error] 8310<a href='/0'>#0</a>: *536 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388580 bytes) in /home/piwik/vhosts/sites-manager.mypiwik.pro/core/DataTable/Renderer/Xml.php on line 202
PHP message: PHP Stack trace:
PHP message: PHP   1. {main}() /home/piwik/vhosts/sites-manager.mypiwik.pro/index.php:0
PHP message: PHP   2. require_once() /home/piwik/vhosts/sites-manager.mypiwik.pro/index.php:47
PHP message: PHP   3. Piwik\FrontController->dispatch() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/dispatch.php:34
PHP message: PHP   4. Piwik\FrontController->doDispatch() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/FrontController.php:86
PHP message: PHP   5. call_user_func_array() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/FrontController.php:576
PHP message: PHP   6. Piwik\Plugins\API\Controller->index() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/FrontController.php:576
PHP message: PHP   7. Piwik\API\Request->process() /home/piwik/vhosts/sites-manager.mypiwik.pro/plugins/API/Controller.php:33
PHP message: PHP   8. Piwik\API\ResponseBuilder->getResponse() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/API/Request.php:217
PHP message: PHP   9. Piwik\API\ResponseBuilder->renderValue() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/API/ResponseBuilder.php:76
PHP message: PHP  10. Piwik\API\ResponseBuilder->handleArray() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/API/ResponseBuilder.php:137
PHP message: PHP  11. Piwik\API\ResponseBuilder->getRenderedDataTable() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/API/ResponseBuilder.php:375
PHP message: PHP  12. Piwik\DataTable\Renderer\Xml->render() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/API/ResponseBuilder.php:230
PHP message: PHP  13. Piwik\DataTable\Renderer\Xml->renderTable() /home/piwik/vhosts/sites-manager.mypiwik.pro/core/DataTable/Renderer/Xml.php:36
PHP message: PHP  14. Piwik\DataTable\Renderer\Xml->renderArray() /home/piwik/vho

And database:

mysql> select count(*) from piwik_site;
| count(*) |
|    20001 |
1 row in set (0,00 sec)
@mattab commented on August 25th 2014 Owner

Ok so it looks like we have some very inefficient code in there... maybe we need a little XHprof analysis to figure out why and maybe we can find solution!

This Pull Request was closed on August 6th 2014
Powered by GitHub Issue Mirror