Chunked not working

S

Seeker-Smith

New member
Nov 17, 2024
11
1
3
#1
I have my php resources set to:
PHP memory_limit 512M
PHP post_max_size 51200M
PHP upload_max_filesize 51200M
PHP max_input_vars 1000
PHP max_execution_time 240

Uploads over 200mb or so were failing so I bought chunked hoping for an easy fix. But it made no difference at all. They still fail at around 200MB.

Chunked setting.
chunk.png

This has only occurred since changing from Apache to Nginx and I'm running php8.3 FPM/FastCGI
 
#2
Hello, although it should work with only one chunk, you could increase the simultaneous chunk files to upload in parallel. Also, do you see any errors in your error_log?
 
#3
JulianD said:
Hello, although it should work with only one chunk, you could increase the simultaneous chunk files to upload in parallel. Also, do you see any errors in your error_log?
Click to expand...

Hello @JulianD & @Seeker-Smith ,

I'd like to join your conversation, as I have the same problem: the add-on suddenly stops working.

I can confirm that there is no PHP error. I have the impression that the resource slices are simply no longer being made. By observing the http requests sent from the web browser when uploading a resource, I no longer see several HTTP requests being sent and no files appearing in directory chunk_uplodads/temp. In the end of the upload, I get a http 413 error (Request Entity Too Large)

I tried to :
  • deactivate the other add-ons.
  • rebuilt and reinstalled the add-on
  • check errors log
  • change add-on parameters

As additional information, here is what I can tell you to help us:
  • Xenforo v2.3.4
  • Apache
  • Add-on Chuncked upload v1.0.3
  • PHP v8.3
    • PHP memory_limit : 128M
    • PHP post_max_size : 20M
    • PHP upload_max_filesize : 20M
I recently updated xenforo to its new version. I can't tell if the problem is related to

I hope you can tell us more @JulianD
Thanks in advance for your help.
 
Last edited:
#4
Since finding a setting in Nginx nginx client_max_body_size that was set to 200m I've been able to increase my uploads but some still fail. I cn't say where chunked is working or not. If I upload a 2 gb file with it on of off it succeeds where another might fail. Since updating the Nginx setting I no longer see the file size error.
 
#5
Seeker-Smith said:
Since finding a setting in Nginx nginx client_max_body_size that was set to 200m I've been able to increase my uploads but some still fail. I cn't say where chunked is working or not. If I upload a 2 gb file with it on of off it succeeds where another might fail. Since updating the Nginx setting I no longer see the file size error.
Click to expand...
In my case, I use Apache. The file upload percentages progress to the end, then I get an error http 413, even for a 500MB file.

I think your first analysis will be useful to others. I'll wait and see if @JulianD has any leads for me.
 
#8
Seeker-Smith said:
Okay with Chunk enabled I can upload to 5G with it enabled and set to:
View attachment 29

a 5.62G file fails. So I'm not sure if this is even functioning. I'm not seeing anything in the server logs.
Click to expand...

Interesting @Seeker-Smith. I've tested it with your settings and unfortunately it doesn't change my problem. I have the feeling that the add-on is simply not being used.

Maybe with your help, I can find out if the add-on works. If you agree to help me, we could give it a try:

Firstly, what is your xenforo version ? Then I have two questions for you :

From administration :
  1. From the administration, go to “Appareance” > Template modification, and you should see a template named “xfmg_media_add_macros”. In the field "Template contents", do you have the data : The requested template could not be found. ?
Uploading file :
  1. Select a file and start uploading it.
  2. From this page, during the upload, press F12 to open the development tools. Go in your “Network” window/tab.
  3. Do you see any requests (representing file splices) being sent while the upload is in progress ?
For your information, I noticed than changing chunk_size parameter can help for bypassing networking/firewall problems ;)

If it's not clear, I'd be happy to help. Thank you in advance for your help.
 
Last edited:
#9
xf_community_dev said:
Interesting @Seeker-Smith. I've tested it with your settings and unfortunately it doesn't change my problem. I have the feeling that the add-on is simply not being used.

Maybe with your help, I can find out if the add-on works. If you agree to help me, we could give it a try:

Firstly, what is your xenforo version ? Then I have two questions for you :

From administration :
  1. From the administration, go to “Appareance” > Template modification, and you should see a template named “xfmg_media_add_macros”. In the field "Template contents", do you have the data : The requested template could not be found. ?
Uploading file :
  1. Select a file and start uploading it.
  2. From this page, during the upload, press F12 to open the development tools. Go in your “Network” window/tab.
  3. Do you see any requests (representing file splices) being sent while the upload is in progress ?
For your information, I noticed than changing chunk_size parameter can help for bypassing networking/firewall problems ;)

If it's not clear, I'd be happy to help. Thank you in advance for your help.
Click to expand...
Sure. My goal with or without is to have a working upload system.

I'm Xenforo 2.3.4, php8.3, Niginx

1) template xfmg_media_add_macros yes I have it but its from Dragonbyte Social groups addon.
2) I have my uploads Nginx and php8.3 now set to 10GB. I just tried a file 5.62GB and it filed with a timeout error. I'm not seeing anything that looks like file slices.
 
  • Like
Reactions: xf_community_dev
#10
Hey guys thanks for reporting the problems you’re having with the addon. I still need some more time to debug it and try to reproduce it with the different environment that have been mentioned here. As soon as I have some progress I’ll keep you yo to date.
Thank you.
 
  • Like
Reactions: xf_community_dev
#11
JulianD said:
Hey guys thanks for reporting the problems you’re having with the addon. I still need some more time to debug it and try to reproduce it with the different environment that have been mentioned here. As soon as I have some progress I’ll keep you yo to date.
Thank you.
Click to expand...

Do not hesitate, let me know if you need something else for helping us.

Thank you in advance !
 
#12
Hello @JulianD & @Seeker-Smith

I managed to get the add-on working again. At least, I manage to have sliced resources in the chunked_upload folder again. I'm currently running further tests.

Here's what I found, template modifications were no longer applied (Appearance > Template modification).

_data/template_modifications.xml​


Origin :
XML: 
<modification type="public" template="helper_attach_upload" modification_key="chunkedUploads_helper_attach_upload" description="Loads the Chunked Uploads javascript file" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="xf\/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0
    <xf:js src="ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
</modification>

<modification type="public" template="xfmg_media_add_macros" modification_key="chunkedUploads_xfmg_media_add_macros" description="Loads the Chunked Uploads javascript file in the XFMG addon" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="xf\/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0<xf:js src="ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
</modification>

New :

XML: 
<modification type="public" template="helper_attach_upload" modification_key="chunkedUploads_helper_attach_upload" description="Loads the Chunked Uploads javascript file" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="vendor/exif-reader/exif-reader.js, xf/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0
<xf:js src="vendor/exif-reader/exif-reader.js, ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
  </modification>
<modification type="public" template="xfmg_media_add_macros" modification_key="chunkedUploads_xfmg_media_add_macros" description="Loads the Chunked Uploads javascript file in the XFMG addon" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="vendor/exif-reader/exif-reader.js, xf/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0
<xf:js src="vendor/exif-reader/exif-reader.js, ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
</modification>

I don't know if xenforo 2.3.4 has changed its helper_attach_upload and xfmg_media_add_macros templates, but they now include exif-reader.js. As a result, the template modification regex was no longer functional.

Can you @JulianD verify these results and keep us informed ?

After a few tests, I can't upload more than a 1.5 GB file. For a 2GB file, the upload completes without error, then when I click on download from XFRM, the following error message appears in a new page : An error occurred while processing your request (There is an HTTP 403 (Forbidden) error).

Thank you in advance for your help.
 
  • Like
Reactions: JulianD
#13
xf_community_dev said:
Hello @JulianD & @Seeker-Smith

I managed to get the add-on working again. At least, I manage to have sliced resources in the chunked_upload folder again. I'm currently running further tests.

Here's what I found, template modifications were no longer applied (Appearance > Template modification).

_data/template_modifications.xml​


Origin :
XML: 
<modification type="public" template="helper_attach_upload" modification_key="chunkedUploads_helper_attach_upload" description="Loads the Chunked Uploads javascript file" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="xf\/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0
    <xf:js src="ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
</modification>

<modification type="public" template="xfmg_media_add_macros" modification_key="chunkedUploads_xfmg_media_add_macros" description="Loads the Chunked Uploads javascript file in the XFMG addon" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="xf\/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0<xf:js src="ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
</modification>

New :

XML: 
<modification type="public" template="helper_attach_upload" modification_key="chunkedUploads_helper_attach_upload" description="Loads the Chunked Uploads javascript file" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="vendor/exif-reader/exif-reader.js, xf/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0
<xf:js src="vendor/exif-reader/exif-reader.js, ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
  </modification>
<modification type="public" template="xfmg_media_add_macros" modification_key="chunkedUploads_xfmg_media_add_macros" description="Loads the Chunked Uploads javascript file in the XFMG addon" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="vendor/exif-reader/exif-reader.js, xf/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0
<xf:js src="vendor/exif-reader/exif-reader.js, ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
</modification>

I don't know if xenforo 2.3.4 has changed its helper_attach_upload and xfmg_media_add_macros templates, but they now include exif-reader.js. As a result, the template modification regex was no longer functional.

Can you @JulianD verify these results and keep us informed ?

After a few tests, I can't upload more than a 1.5 GB file. For a 2GB file, the upload completes without error, then when I click on download from XFRM, the following error message appears in a new page : An error occurred while processing your request (There is an HTTP 403 (Forbidden) error).

Thank you in advance for your help.
Click to expand...
Have you reported this to the Xenforo team?
 
#14
Seeker-Smith said:
Have you reported this to the Xenforo team?
Click to expand...
Hello @Seeker-Smith,

No, I haven't contacted the XenForo team, as they're not responsible for upgrading community extensions (see screenshot below).

Capture d’écran 2024-12-09 142025.png

To find out if your extension needs to be upgraded to xenforo 2.3.4, you'll need to try out this version. The problem may be linked to a different configuration on my side. Could you please check?

Thanks in advance.
 
#15
xf_community_dev said:
Hello @Seeker-Smith,

No, I haven't contacted the XenForo team, as they're not responsible for upgrading community extensions (see screenshot below).

View attachment 31

To find out if your extension needs to be upgraded to xenforo 2.3.4, you'll need to try out this version. The problem may be linked to a different configuration on my side. Could you please check?

Thanks in advance.
Click to expand...
You indicated that you edited helper_attach_upload and xfmg_media_add_macros templates I assumed there was an error Xenforo should be aware of.
 
#16
xf_community_dev said:
Hello @JulianD & @Seeker-Smith

I managed to get the add-on working again. At least, I manage to have sliced resources in the chunked_upload folder again. I'm currently running further tests.

Here's what I found, template modifications were no longer applied (Appearance > Template modification).

_data/template_modifications.xml​


Origin :
XML: 
<modification type="public" template="helper_attach_upload" modification_key="chunkedUploads_helper_attach_upload" description="Loads the Chunked Uploads javascript file" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="xf\/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0
    <xf:js src="ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
</modification>

<modification type="public" template="xfmg_media_add_macros" modification_key="chunkedUploads_xfmg_media_add_macros" description="Loads the Chunked Uploads javascript file in the XFMG addon" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="xf\/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0<xf:js src="ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
</modification>

New :

XML: 
<modification type="public" template="helper_attach_upload" modification_key="chunkedUploads_helper_attach_upload" description="Loads the Chunked Uploads javascript file" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="vendor/exif-reader/exif-reader.js, xf/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0
<xf:js src="vendor/exif-reader/exif-reader.js, ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
  </modification>
<modification type="public" template="xfmg_media_add_macros" modification_key="chunkedUploads_xfmg_media_add_macros" description="Loads the Chunked Uploads javascript file in the XFMG addon" execution_order="10" enabled="1" action="preg_replace">
    <find><![CDATA[#<xf:js prod="vendor/exif-reader/exif-reader.js, xf/attachment_manager-compiled.js"[^>]*?\/>#i]]></find>
    <replace><![CDATA[$0
<xf:js src="vendor/exif-reader/exif-reader.js, ChunkedUploads/xf/attachment_manager.js" min="1" addon="ChunkedUploads" />]]></replace>
</modification>

I don't know if xenforo 2.3.4 has changed its helper_attach_upload and xfmg_media_add_macros templates, but they now include exif-reader.js. As a result, the template modification regex was no longer functional.

Can you @JulianD verify these results and keep us informed ?

After a few tests, I can't upload more than a 1.5 GB file. For a 2GB file, the upload completes without error, then when I click on download from XFRM, the following error message appears in a new page : An error occurred while processing your request (There is an HTTP 403 (Forbidden) error).

Thank you in advance for your help.
Click to expand...
Thanks for reporting this. I have fixed this and released a new version. Let me know if this works for you.
 
#17
Seeker-Smith said:
You indicated that you edited helper_attach_upload and xfmg_media_add_macros templates I assumed there was an error Xenforo should be aware of.
Click to expand...
Hello,

Sorry @Seeker-Smith, I confused you with @JulianD (The author of the add-on). I now understand your comment about contacting the Xenforo team.

JulianD said:
Thanks for reporting this. I have fixed this and released a new version. Let me know if this works for you.
Click to expand...

I'll try and give you some feedback. Thank you for your update !
 
#19
No joy for me. I've tried uploading a file a bit over 5GBwith the settings in the attachment and it fails at around 60% with nothing in the server error log.
 

Attachments

  • attach.png
    attach.png
    220.3 KB · Views: 3
You must log in or register to reply here.
Top Bottom
Back
Menu