Hi Bobwieser, I'm not a professional geologist, so take this post with a grain of sand (or quartz, har har.), but I know a bit about it.
Just thinking through your process here.... so, you're trying to emulate fireclay.
In "fireclay" you've got "fire" and you have "clay".
The "fire" part is referring to the mineral composition and the "clay" part is referring to particle size. Forgive my simplistic explanation, I'm not trying to be an ass, just stick with me!
The reason I asked about the feldspar is because you'll basically find two types. Orthoclase feldspar (K-feldspar) and plagioclase feldspar (P-feldspar). P-feldspar is often pink in color, the more pink, the more pure. If you want P-feldspar and you've got really pink granite, you're in luck. In this case you want K-feldspar because you're looking for the kaolinite minerals and aluminum oxides that fireclay have. Problem is, white or gray granites might be really pure K-feldspar or rather impure P-feldspar. I would lean toward thinking what you have there is probably K-feldspar or at least close enough to work with.
On to step two, mineral size. Clays are extremely fine, <2 nanometers. Granite comes in many forms depending on how deep inside the earth or quickly it cooled. Regardless of this, by nature the crystal size in granites are way bigger than clays, not to mention they have about 0 organic matter.
So if you want to use it I would be inclined to have a couple of thoughts. It might work, so far it looks like the color might be right. Determining particle size is a bit of a process. The ASTM has a standard for it and it only requires about $15 worth of equipment and some time. Even easier, you can mix some of your material in a jar of water and see how quickly it settles out. I'm not sure how fine this dust might be cut from the saws that your supplier is using, but I would go out on a limb and think it's not fine enough but it still might work by incorporating a binder like fire cement or some ash.
Fireclay isn't so much for insulation as it is for being able to withstand the heat. The vermiculite or perlite is for insulation, and you can get a big ass bag of it at any big-box hardware store for about $10, so I wouldn't worry about using up your perlite.
Now for the frustrating part. I sometimes get irritated going on these forums asking questions and most responses are "I don't know, try it and see". Fact of the matter is that most of this stuff is experimental and I have to tell you: I don't know, try it and see.
I might start with something simple like 45% perlite, 45% granite dust, 10 % firecement (by volume) and go from there. Or maybe try 40% perlite, 40% granite, and 20% ash. I bet it will work, however what the "right" ratio is....... I dunno? I wouldn't even be afraid to try adding some Portland cement. Honestly, you can make this things out of mud and they will last indefinitely.
If it's something you want to last for a long time, I would consider investing in some firecement for the critical areas like the feed.
Good luck!
"Knowledge is power. Arm yourself."