In The Name of Allah, The Most Merciful, The Bestower of Mercy.

Allah stated that Prophet, Ibrahim, peace be upon him, said to his people:

وَڪَيۡفَ أَخَافُ مَآ أَشۡرَڪۡتُمۡ وَلَا تَخَافُونَ أَنَّكُمۡ أَشۡرَكۡتُم بِٱللَّهِ مَا لَمۡ يُنَزِّلۡ بِهِۦ عَلَيۡڪُمۡ سُلۡطَـٰنً۬ا‌ۚ فَأَىُّ ٱلۡفَرِيقَيۡنِ أَحَقُّ بِٱلۡأَمۡنِ‌ۖ إِن كُنتُمۡ تَعۡلَمُونَ

And how should I fear those whom you associate in worship with Allah (though they can neither benefit nor harm), while you fear not that you have joined in worship with Allah things for which He has not sent down to you any authority. (So) which of the two parties has more right to be in security? If you but know.” It is those who believe (in the Oneness of Allah and worship none but Him Alone) and confuse not their belief with Zulm (wrong i.e. by worshipping others besides Allah), for them (only) there is security and they are the guided. [Al-An’aam. 81-82]

وَڪَيۡفَ أَخَافُ مَآ أَشۡرَڪۡتُمۡ

Why should I fear that which you ascribe as partners to [Allah]? – Meaning, when its state is that of inability and cannot benefit.

وَلَا تَخَافُونَ أَنَّكُمۡ أَشۡرَكۡتُم بِٱللَّهِ مَا لَمۡ يُنَزِّلۡ بِهِۦ عَلَيۡڪُمۡ سُلۡطَـٰنً۬ا‌ۚ

“whilst you do not fear ascribing partners to Allah for which He has not sent down to you any authority”..

Meaning: you do so (i.e. worship false dieties) solely due to following desires!

فَأَىُّ ٱلۡفَرِيقَيۡنِ أَحَقُّ بِٱلۡأَمۡنِ‌ۖ إِن كُنتُمۡ تَعۡلَمُونَ

(So) which of the two parties has more right to be in security? If you but know?

Then (Allah) differentiated the two parties and stated:

ٱلَّذِينَ ءَامَنُواْ وَلَمۡ يَلۡبِسُوٓاْ إِيمَـٰنَهُم بِظُلۡمٍ أُوْلَـٰٓٮِٕكَ لَهُمُ ٱلۡأَمۡنُ وَهُم مُّهۡتَدُونَ

It is those who believe (in the Oneness of Allah and worship none but Him Alone) and confuse not their belief with Zulm (wrong i.e. by worshipping others besides Allah), for them (only) there is security and they are the guided] -Meaning, safe from fear, punishment and misery, and they will be guided to the straight path. [1]

We shall cast terror into the hearts of those who disbelieve, because they ascribe partners to Allah, for which He had sent no authority. [Aal Imran. 151]

Imam Ibn Al-Qayyim, may Allah have mercy upon him, said:

The level of Shirk determines the level of fear, and the one who ascribes a partner with Allah has the greatest fear and terror. Those who believe and do not mix their belief with Shirk have safety, guidance and success and the one who ascribes a partner to Allah has fear, misguidance and wretchedness”. [2]

[1]An Excerpt From Tafseer As-Sadi

[2] An Excerpt from Iddat As-Saabireen 166-167