whitebear whitebear - 4 months ago 13
PHP Question

PHP Fatal error: Allowed memory size of 268435456 bytes exhausted on NormalizerFormatter.php on line 62

I am trying to insert around 20000 records in mysql by doctrine2

foreach ($getIds as $id){// 20000 Ids
$twNode = new TwNode();
$twNode->setTwId($id);
$twNode->addAcFriend($this->twAccount);
$this->twAccount->addFollower($twNode);
$this->em->persist($twNode);
$this->em->persist($this->twAccount);
$this->em->flush();
$this->output->write('W');
$followersWrite++;
}
}


However this error happnes, how can I avoid this Allowed memory error?

PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes) in /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php on line 62
PHP Stack trace:
PHP 1. {main}() /Users/whitebear/httproot/myapp/app/console:0
PHP 2. Symfony\Component\Console\Application->run() /Users/whitebear/httproot/myapp/app/console:22
PHP 3. Symfony\Bundle\FrameworkBundle\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:106
PHP 4. Symfony\Component\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:83
PHP 5. Symfony\Component\Console\Command\Command->run() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:200
PHP 6. Acme\TopBundle\Command\twCrawlerCommand->execute() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:242
PHP 7. Acme\TopBundle\Command\twCrawlerCommand->initialRun() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:430
PHP 8. EntityManager52ee6dcbdb2bb_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:284
PHP 9. Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/app/cache/dev/jms_diextra/doctrine/EntityManager_52ee6dcbdb2bb.php:402
PHP 10. Doctrine\ORM\UnitOfWork->commit() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:389
PHP 11. Doctrine\ORM\UnitOfWork->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:335
PHP 12. Doctrine\ORM\Persisters\BasicEntityPersister->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:952
PHP 13. Doctrine\DBAL\Statement->execute() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/BasicEntityPersister.php:279
PHP 14. Doctrine\DBAL\Logging\LoggerChain->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php:161
PHP 15. Symfony\Bridge\Doctrine\Logger\DbalLogger->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Logging/LoggerChain.php:54
PHP 16. Symfony\Bridge\Doctrine\Logger\DbalLogger->log() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:80
PHP 17. Monolog\Logger->debug() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:102
PHP 18. Monolog\Logger->addRecord() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:423
PHP 19. Monolog\Handler\AbstractProcessingHandler->handle() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:244
PHP 20. Monolog\Formatter\LineFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:35
PHP 21. Monolog\Formatter\NormalizerFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:43
PHP 22. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:40
PHP 23. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88
PHP 24. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:70
PHP 25. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes) in /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php on line 62

Call Stack:
0.0002 229968 1. {main}() /Users/whitebear/httproot/myapp/app/console:0
0.0126 2063376 2. Symfony\Component\Console\Application->run() /Users/whitebear/httproot/myapp/app/console:22
0.0144 2256968 3. Symfony\Bundle\FrameworkBundle\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:106
0.1503 11751600 4. Symfony\Component\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:83
0.1518 11753384 5. Symfony\Component\Console\Command\Command->run() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:200
0.1521 11758040 6. Acme\TopBundle\Command\twCrawlerCommand->execute() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:242
0.2409 20673112 7. Acme\TopBundle\Command\twCrawlerCommand->initialRun() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:430
101.5883 235069104 8. EntityManager52ee6dcbdb2bb_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:284
101.5883 235069200 9. Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/app/cache/dev/jms_diextra/doctrine/EntityManager_52ee6dcbdb2bb.php:402
101.5883 235069280 10. Doctrine\ORM\UnitOfWork->commit() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:389
107.0744 259211408 11. Doctrine\ORM\UnitOfWork->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:335
107.1128 259277872 12. Doctrine\ORM\Persisters\BasicEntityPersister->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:952
109.1604 267877360 13. Doctrine\DBAL\Statement->execute() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/BasicEntityPersister.php:279
109.1604 267877408 14. Doctrine\DBAL\Logging\LoggerChain->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php:161
109.1604 267877408 15. Symfony\Bridge\Doctrine\Logger\DbalLogger->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Logging/LoggerChain.php:54
109.1604 267877592 16. Symfony\Bridge\Doctrine\Logger\DbalLogger->log() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:80
109.1604 267877592 17. Monolog\Logger->debug() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:102
109.1604 267877640 18. Monolog\Logger->addRecord() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:423
109.1604 267878968 19. Monolog\Handler\AbstractProcessingHandler->handle() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:244
109.1604 267878968 20. Monolog\Formatter\LineFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:35
109.1604 267878968 21. Monolog\Formatter\NormalizerFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:43
109.1604 267878968 22. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:40
109.1604 267878968 23. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88
109.1605 267881872 24. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:70
109.1605 267881872 25. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88


Thanks to your comment, I understood it is relavant with the memory comsumed by $this->em

so, I put the codes to unset $this->em every 5000 times;

if ($k > 1000){
$this->em->flush();
unset($this->em);
$this->em = $this->getContainer()->get('doctrine')->getManager();
}


but still in vain, I think I should unset the origin of $this->getContainer()->get('doctrine')->getManager();

is it possible? or am I right?

Answer

You may want to read Doctrine 2 Batch Processing. They are using this pattern:

// $em instanceof EntityManager
$batchSize = 20;
for ($i=1; $i<=10000; ++$i) {
     $obj = new MyEntity;
     $obj->setFoo('...');
     // ... set more data
     $em->persist($obj);
     if (($i % $batchSize) == 0) {
         $em->flush();
         $em->clear();
    }
}
$em->flush();

As I understand it - after persisting, doctrine holds all mapped objects in memory in identity map. So memory requirements grow all the time with every new object. If you do clear(), it will detach all managed entities and if you have no references to them, they will be garbage-collected.

Beware: it will detach all other entities you are using. Or you can detach only those inserted objects $em->detach($twNode); // detach from Doctrine, so that it can be GC'd immediately

Edit: added flush() after for loop to ensure saving all entities even when entity count is not divisible by batch size, as suggested in comment

Comments